Comments on: Ranked Set Retrieval

By: Daniel Tunkelang

Daniel Tunkelang — Tue, 11 Aug 2009 00:49:03 +0000

Le, you’re correct that the system can mix up the set retrieval and ranked retrieval modes–indeed, that’s the whole point of this model. But forcing a total ordering on the individual documents defeats the point of the model. The idea is to use such a model to optimize for interaction.

For example, let’s say that you can return 10 result elements in response to a query, where a result element is a set of 1 or more documents. What should those 10 result elements be? This is the sort of question that invites a framework like the one proposed by Norbert Fuhr.

By: Le Zhao

Le Zhao — Mon, 10 Aug 2009 23:18:19 +0000

Hi Daniel,

As you said, there is nothing with tiered/ranked set retrieval that prevents you from ranking the documents within each tier.

For example, you can always rank the documents within each set, say according to the ranking formula used in ranked retrieval. Then, if the result sets are ranked relative to all the other sets, you get a global ranked list of documents, even for your tiered method, and you can directly compare the two approaches.

In a flexible search engine e.g. Indri, the independence between filtering (ranking sets) and ranking documents is very clear. There are two kinds of query operators, the Boolean filtering kind for tiered ranking, and the scoring kind, for ranking. Users can mix these two kinds of operators in their queries. However, the result will always be a single ranked list.

The only problem might be result clustering. As there is no ranking across the different clusters, and that changes the result presentation, thus, might be difficult to directly compare with ranked retrieval. I guess this is where the problem really is? Not the previous set ranking case?

By: Norbert Fuhr’s Probability Ranking Principle for Interactive Information Retrieval | The Noisy Channel

Mon, 10 Aug 2009 14:06:38 +0000

[...] Fuhr’s model. I suspect I was trying to back into a similar approach in my post on “Ranked Set Retrieval” a few months ago. But, even if I’d try to formalize my proposal, I don’t believe [...]

By: FXPAL Blog » Blog Archive » Models of interaction, part 1

FXPAL Blog » Blog Archive » Models of interaction, part 1 — Fri, 06 Mar 2009 18:44:36 +0000

[...] of retrieval. The two are complementary: for example, recently Daniel Tunkelang posted about using sets rather than ranked lists as a way of representing search results. This has implications on one hand [...]

By: Daniel Tunkelang

Daniel Tunkelang — Thu, 05 Mar 2009 13:52:41 +0000

Well, there’s no reason you can’t rank the list of results returned in a faceted search engine. We do that at Endeca all the time. But that’s still a sequence of singleton sets: ranked retrieval with the implied appended subset of non-matching documents at the end.

But, since we return facet values associated with sets of documents, you could say that each facet value represents a subset of the result set, and that our ordering of facet values is a more general kind of ranked set retrieval. In fact, you can even see it recursively, since selecting one of these values will lead to another such set.

But we have never approached the subject with the theoretical rigor needed for something as ambitious as I outlined above. I don’t think anyone has.

By: Max L. Wilson

Max L. Wilson — Thu, 05 Mar 2009 10:44:39 +0000

bah - i just wrote a nice long comment and it got lost. It was along the lines of: This post has been stewing in the back of my head... yada yada... Bringing the two communities has been a long time coming. Its on the way... I agree with your post... then: There was an interesting paper at SearchSolutions2008 in London last year. It presented a faceted system they had built that applied weightings to the index, rather than performing set retrieval. So it was facets, but returning a ranked list, in conjunction with keywords. It was presented by Lemur Consulting.

By: Daniel Tunkelang

Daniel Tunkelang — Tue, 03 Mar 2009 16:41:23 +0000

Gene, I like scatter/gather, but the question I think it leaves is how you compare it to a ranked retrieval approach. For that matter, how could a system decide between the two for the same query? I feel like there’s a false dichotomy between clustering / query refinement / set-oriented approaches and ranked ones that is impeding research progress. Indeed, I think this is a major issue for the HCIR community, at least with respect to finding common ground with IR traditionalists.

By: Daniel Tunkelang

Daniel Tunkelang — Tue, 03 Mar 2009 16:36:53 +0000

Jeremy, you’re right, at least in that his future work suggestion on optimal cluster retrieval (section 8.3.1) is very much in this vein. I’m a fan of Fernando’s work, though I’ll confess I’ve never read his dissertation from cover to cover. Do you know if he or anyone else has followed up on these ideas?

Actually, Fernando, maybe you’re reading this yourself and can chime in.

By: Gene Golovchinsky

Gene Golovchinsky — Tue, 03 Mar 2009 16:36:13 +0000

This approach makes sense. In fact, it reminds me a bit of scatter/gather. One HCI concern is people’s expectations. Search results will need to be presented in a palatable way. Perhaps this approach may work better for injecting search results into other tasks rather than serving the up raw.

By: jeremy

jeremy — Tue, 03 Mar 2009 16:28:41 +0000

I think Fernando Diaz did some interesting work, if not directly on the things you are talking about, at least one something related. Check out his 2008 dissertation:

http://ciir.cs.umass.edu/~fdiaz/