In library science, the first question to ask is not how to formulate a query, but where to look for the information. Cheers.

I’ve blogged about high recall (and here and here).

Our NIH grant is focused on high recall linkage of the research literature to gene databases (and other databases). In the gene case, for any given release of the databases, the queries are fixed — they’re just the genes in the Entrez-Gene database. And we just want to find articles that mention them.

A serious issue is results diversity when trying to formalize useful recall. Both micro- and macro-averages are misleading, as is the pooled evaluation strategy of TREC, which tends to overestimate true recall (by assuming each true positive was found by at least one team in the top 1000 results).

It’s very easy to find some genes and very hard to find others. We can find all gazillion mentions of “p53″ with high precision, but have a harder time with the mouse gene “at”. And that applies across aliases within genes, too; Alpha-1-antichymotrypsin is easy to find when referred to as “SERPINA3″, but much harder when called “ACT” (there’s lots of all caps titles, etc. in MEDLINE).

Your point (I thought) was that if in the historical literature there is a proliferation, then we do not have a problem. (It would be interesting to do a survey and see whether metrics “proliferate”.)

My counterpoint is that we don’t have to do a survey, because we can proliferate the metrics ourselves. By writing papers that propose, and utilize, new metrics.

Both recall and precision aside, one of the things I learned very early on in my IR career is that there is a strong (at least theoretical) willingness in the general community to accept any paper that uses a non-traditional evaluation metric, as long as that metric is justified and objective (not just made up to fit the data).

Granted, not every single reviewer feels this way, but I’ve had enough private conversations over the years to know that a lot of people do. In fact, I’ll venture out on a limb and make the observation that the younger the researcher, the more dogmatic he or she tends to be about the evaluation metric used. I’ve had many more conversations with IR “old timers” in which they’ve expressed willingness to let metrics proliferate.

And hypothetically, suppose almost no one allowed metrics to proliferate. If that were the case, it doesn’t mean we can’t do good science and use good metrics anyway. Yes, it might be much more difficult to get papers accepted, and to get funding, etc. But that’s a constant problem that we face either way.. convincing others of the value of our work.

But luckily, that’s not the case. I do find a willingness in the IR community to accept papers with non-traditional metrics.

Beyond that I think we still are dealing with context. Even the results are contextual.

My point was precisely as stated. If we have a proliferation of metrics, then we are not likely to have a problem.

Thus, in any field, not just IR, it would be interesting to do surveys to see whether people use and develop a wide range of metrics.

Then, if they don’t, we should investigate and determine whether it is a problem.

I think it is an interesting, generic, research program.

I don’t quite understand what you mean by doing a survey. Mine was a proscriptive suggestion, not a descriptive.

But the suggestion does have precedent. In the early days of TREC (early-mid 90s), there wasn’t just one way of presenting precision or recall. You have everything from MAP, to Prec@n, to R-prec, to recall@1000, to interpolated precision/recall (e.g. prec at 0.0 interpolated recall, at 0.1, etc.) .. and then later GMAP, and ndgc, and.. and..

And a lot of the early studies didn’t just report one number, one metric. A lot of the early studies reported *lots* of metrics.

It is only relatively recently.. really since the popularization of the web and web search engines, that the number of reported metrics per paper seems to have decreased.

I still say, go with all of ‘em. Give the biggest, most robust picture available.

And a lot

I think it might be a bit more difficult to execute with rigor. I’d be interested in any reference.

All metrics are only approximations. And as long as we don’t forget that, the approach that I favor is to let metrics proliferate.

It would be interesting to do a survey and see whether metrics “proliferate”. If they do, then I agree we do not have a problem.

You can still do multi-shot query evaluation and measure it in terms of precision and recall. You might have to ask users to trawl through real results until they’ve completed their task, but at the end of the day the entire set of information that the user has examined can be thought of as a sequence or a list. And with that list, you can perform precision and recall calculations.

Well, you might have to change the name of the metric to “viewed precision” and “viewed recall”. But the overall concept still applies.

However, even if you do measure it this way, there is still the issue of how well recall and/or precision approximate task completion (aka “information need satisfaction”), which we all agree is what we’re really after.

Yes, there is a danger of reifying the evaluation metric, whether that metric is precision, recall, ndcg or whatever. But that temptation to reify is going to be true of any and every evaluation metric, no matter what it is. All metrics are only approximations. And as long as we don’t forget that, the approach that I favor is to let metrics proliferate. The more metrics you use, the more approximations, the more samples points you have, and that can only be a good thing.

That is, instead of being “against recall” because it is unclear whether one means cardinality, coverage, density, etc…why not be pro-recall, pro-cardinality, pro-coverage, pro-density, pro-precision, pro-etc.?

I don’t think that database people ever assume that the user will only issue one query. In application such as OLAP, you actually expect a stream of queries.