Comments on: Precision and Recall

By: In Defense Of Recall | The Noisy Channel

In Defense Of Recall | The Noisy Channel — Fri, 17 Jul 2009 18:08:52 +0000

[...] abstract from this concrete example and consider what I characterize as the information availability problem, where the information seeker faces uncertainty as to [...]

By: Bringing the Noise to Technology Review | The Noisy Channel

Bringing the Noise to Technology Review | The Noisy Channel — Tue, 23 Jun 2009 13:56:31 +0000

[...] this issue, the recent issue of IEEE Computer with a “beyond search” theme, and of course the Google / Bing marketing battles, it’s great to see an [...]

By: Daniel Tunkelang

Daniel Tunkelang — Thu, 19 Mar 2009 01:39:07 +0000

Abdur, that’s a point I hadn’t considered. I suppose I shouldn’t make a blanket assumption that everyone is teaching to the test, so to speak. Still, my experience in the non-academic world suggests a need to invest more effort on the recall side, particularly for applications that offer any kind of summarization of result sets.

In any case, thanks for the encouragement. I’m looking forward to seeing where you and your team go with Twitter search.

By: Abdur

Abdur — Wed, 18 Mar 2009 21:41:05 +0000

Daniel I use F1 often. I am not sure people ignore the trade offs just don’t report them in strict IR papers because they are often selling an algorithm that optimizes one. Look at the ML research where those trad-offs are considered and the balance of those competing goals are articulated much more. Keep up the good posts…

By: Daniel Tunkelang

Daniel Tunkelang — Wed, 18 Mar 2009 16:09:00 +0000

Ellen, as you point out, it's a thorny problem. I'm actually partial to using a game rather than a batch procedure for testing recall, as per my post on "Games With an HCIR Purpose?". But it would take a lot of work and politicking to implement such a research program.

By: jeremy

jeremy — Wed, 18 Mar 2009 15:59:30 +0000

Ellen, I’ve been beating that drum in the online world for at least 5 or 6 years now. Longer in the offline world (starting about 11 years ago I was doing music information retrieval, and that’s a perfect example of a vertical in which recall is the more important metric.)

Maybe it’s just that our voices are too small, too far out in the wilderness of the web landscape? But we do exist

By: Ellen Voorhees

Ellen Voorhees — Wed, 18 Mar 2009 13:23:32 +0000

I find it refreshing to see a reminder that precision is not the only factor that matters in search from someone other than me. Indeed, most of the “discussions” I now have regarding retrieval evaluation involve the claim that MAP is too *recall*-oriented to be a useful measure.

One of the themes of the TREC legal track, which focuses on supporting legal discovery, has been fair evaluation of recall-oriented searches in massive collections. It is a thorny problem that remains mostly unsolved for the general case where you would like reusable test collections as a result.

Ellen