I was extremely flattered to receive an email from Ellen Voorhees responding to my post about Nick Belkin’s keynote. Then I was a little bit scared, since she is a strong advocate of the Cranfield tradition, and I braced myself for her rebuttal.
She pointed me to a talk she gave at the First International Workshop on Adaptive Information Retrieval (AIR) in 2006. I’d paraphrase her argument as follows: Nick and others (including me) are right to push for a paradigm that supports AIR research, but are being naïve regarding what is necessary for such research to deliver effective–and cost-effective–results. It’s a strong case, and I’d be the first to concede that the advocates for AIR research have not (at least to my knowledge) produced a plausible abstract task that is amenable to efficient evaluation.
To quote Nick again, it’s a grand challenge. And Ellen makes it clear that what we’ve learned so far is not encouraging.