I think measures that factor in time as well as the amount of relevant material retrieved will be useful to evaluate atleast the first two criteria you have mentioned. You might want to look at papers by Mika Kaki, especially this one. Another interesting read would be this SIGIR 2007 poster on correlations between IR measures and user satisfaction.

But there is a difference between a task goal and a query goal. As I perform a sequence of queries to meet my information need, I’m not necessarily concerned with retrieving relevant documents on each query. Rather, I’d like to be learning how to improve my query strategy to ultimately complete my task as successfully as possible.

So yes, relevance is a necessary criterion. But not necessarily in the way that the Cranfield/TREC experiments measure it.

I agree that this is part of the problem, Daniel. That’s the reason why the relevance assessor is the same person who has the information need and crafts the query/topic. There is no absolute (topical) relevance. A measure like AP is evaluating how well a system returns relevant documents according to one person’s idea of relevance at the time the assessment was made. There always are and always will be disagreements between assessors, but that’s the point (and what makes our job interesting)!

Assessors are typically not asked to penalize redundant documents, documents with mis-spellings or grammatical errors, documents from known unreliable sources, etc. These are all facets of what a user may consider useful or not… but they are not traditionally considered aspects of topical relevance.

Relevance is a necessary but not sufficient criterion for utility.

Still, I can’t leave your assertion that “retrieving relevant documents is undoubtedly the leading indicator of the effectiveness of a system” unchallenged. My concern is not just the subjective nature of relevance assessment–if that were the only issue, then we could see the average judgment of assessors as an “objective” relevance measure. The problem is the subjective nature of relevance itself. Ultimately, relevant information is just whatever information I want / need right now.

What implication does this have? Hard to say for sure, but its roughly the difference in systems having a relevant document at the top two ranks 75% of the time vs. 30% of the time (assuming 1-2 relevant documents per query). AP, like Reciprocal Rank, is very sensitive at the high end of the range, so a difference in MAP of 0.85 and 0.95 is due to very small perturbations in the rankings. Differences between 0.25 and 0.35, on the other hand, require a much larger shuffle of the document ranks.

The user Turpin user study you cite, “User performance versus precision measures for simple search tasks”, is interesting but FAR from a definitive nail in the coffin for MAP. Two major “flaws” (I really hesitate to use that word) I see — first, the MAP of the systems they’re testing are in the range of 0.7 – 0.9 (if I remember correctly), and they aim to show that there’s no noticeable difference in the systems from the users’ point of view. This MAP range is really well above the range of MAP scores we see in TREC style evaluations, which is usually closer to 0.3-0.5 for most tasks. Second, measuring user satisfaction in a controlled lab environment with search tasks dictated by someone other than the users is… well… unsatisfying. I’m sure I don’t need to convince you of that.

The real issue, IMHO:

Relevance is not only subjective, but its also just part of the picture when talking about utility & user satisfaction. The Cranfield/TREC methodology allows for subjectivity — queries and relevance judgements are given by a single person, with the intent that a relevant document is relevant for that assessor, not absolutely relevant for everyone. However, Cranfield and evaluation measures like MAP only look at relevance, not all the other factors that make a system truly useful: authority, diversity, recency, and many others.

It has been useful for us IR researchers to focus on relevance — retrieving relevant documents is undoubtedly the leading indicator of the effectiveness of a system. Although I don’t think we’ve solved this problem, we all need to be aware that that is not the only criteria for a successful IR system.