Month: June 2010

I’m looking forward to attending SIGIR 2010 in a few weeks and particularly to the SimInt 2010 Workshop on the Automated Evaluation of Interactive Information Retrieval. I hope I get to see a little bit of the city of Geneva, but mostly I’m excited to spend the greater part of a week immersed in the global information retrieval community.

Of course I’ll blog about the conference, though I can’t promise it will be at quite the level of detail I managed last year. Also, I’m glad that SIGIR is continuing to have an industry track, and I am impressed with the program that David Harper and Peter Schäuble have put together. Needless to say, I’m glad to not have the stress of being an organizer this year! Though I’ll put in an early plug for CIKM 2011 in Glasgow, where I’ll be organizing the industry track with former co-worker Tony Russell-Rose.

Some SIGIR papers that caught my attention in the program:

• Predicting Search Frustration
  Henry Feild, James Allan (University of Massachusetts Amherst), Rosie Jones (Yahoo! Labs)
  (looks like a follow-up to the first two authors’ HCIR 2009 paper on Modeling Searcher Frustration)
• Relevance and Ranking in Online Dating Systems
  Fernando Diaz, Donald Metzler, Sihem Amer-Yahia (Yahoo! Labs)
• On Statistical Analysis and Optimization of Information Retrieval Effectiveness Metrics
  Jun Wang, Jianhan Zhu (University College London)
• Is the Cranfield Paradigm Outdated? (keynote)
  Donna Harman (NIST)
• Interactive Retrieval Based on Faceted Feedback
  Lanbo Zhang, Yi Zhang (University of California at Santa Cruz)
• Do User Preferences and Evaluation measures Line Up?
  Mark Sanderson, Monica Lestari Paramita, Paul Clough, Evangelos Kanoulas (University of Sheffield)
• Human Performance and Retrieval Precision Revisited
  Mark D. Smucker, Chandra Prakash Jethani (University of Waterloo)

As for the SimInt workshop, it aims “to explore the use of Simulation of Interactions to enable automated evaluation of Interactive Information Retrieval Systems and Applications.” I’m very excited about this attempt to bridge the gap between TREC/Cranfield and IIR/HCIR through simulation. Props to Leif Azzopardi, Kal Järvelin, Jaap Kamps, and Mark Smucker for organizing it!

If you’re planning to attend SIGIR, please give me a shout! I plan to be there for the entire conference, and you’ll probably find me at the Google booth during some of the coffee breaks.

Prediction markets were all the rage a few years ago, two of the most notable being the Iowa Electronic Market forecasting electoral results and the now defunct Tradesports offering a similar platform for betting on sports events. There was even a proposal to have the US government run a prediction market for terrorist attacks.

In a prediction market, any event with a quantifiable (e.g., binary) outcome can be converted into an asset. At any given time, the asset value corresponds to the market prediction of the probability of the outcome. Just as in any security market, participants determine the value through their buying and selling actions. In principle, this framework allows any event with a quantifiable outcome to be predicted by a marketplace.

But, at least from my vantage point, prediction markets have not had a broad impact on decision making, despite all of the “anys” in the previous paragraph. Outside of political forecasting and sports gambling (and of course finance itself), I’m not aware of any groups outside of academia that invest significantly in the use of  prediction markets. Sure, there’s the Hollywood Stock Exchange that applies the fantasy sports concept to the movie industry and even startup Empire Avenue that aspires to generalize this idea even further into an “online influence stock exchange”. Still, I think it’s safe to say that prediction markets have had limited traction to date.

Many people do, however, believe that we can harness the wisdom of crowds. In particular, we as consumers rely on reviews and recommendations to inform our decisions about what to buy, read, etc. Because those decisions have financial implications for sellers, the world of online reviews has an adversarial element, where review systems face manipulation by those who would shill their own products or services. As a result, it is never clear how much we as consumers should trust the reviews we read to be sincere, let alone useful.

Which brings me back to prediction markets. Unlike most venues for soliciting collective opinion, prediction markets offer a strong incentive for accuracy. Betting on whether readers will like a book is quite different than simply offering a review that asserts an opinion without any risk to the person making the assertion. It is possible to manipulate a prediction market (e.g., by flooding it with high bets), but research suggests that such manipulations are short-lived and in fact expose the manipulator to significant financial risk when the price re-stabilizes.

So why don’t we use prediction markets instead of relying on reviews and recommendations? Perhaps we should, and it’s just a matter of time until entrepreneurs build successful businesses around this idea. But I suspect that much of the value of user-generated content today comes from contributors not thinking in market terms. While using prediction markets could solve the problem of shill reviews, it might also scare off the altruists.

Still, it seems to me that we should look for more opportunities to incent accuracy. Even altruistic reviewers have an interest in establishing their credibility, at least if that credibility determines the propagation of they opinions they share (perhaps I’m conflating altruism with egotism). The challenge may be to implement a marketplace that deals in the social currency of reputation than the hard currency of cash–while avoiding the sort of virtual currency that many people see as meaningless.

Can we obtain the benefits of market dynamics and still take advantage of the less rational motivations that drive some of the best online reviews today? I hope there are people who feel incented to work on this problem!

Some previous posts for further reading:

• Thoughts About Online Reputation
• Payola? There’s An App For That!
• Are Links A Distraction? (please let me know if you like my attempt to keep links outside the main flow)