Category: General

Prediction markets were all the rage a few years ago, two of the most notable being the Iowa Electronic Market forecasting electoral results and the now defunct Tradesports offering a similar platform for betting on sports events. There was even a proposal to have the US government run a prediction market for terrorist attacks.

In a prediction market, any event with a quantifiable (e.g., binary) outcome can be converted into an asset. At any given time, the asset value corresponds to the market prediction of the probability of the outcome. Just as in any security market, participants determine the value through their buying and selling actions. In principle, this framework allows any event with a quantifiable outcome to be predicted by a marketplace.

But, at least from my vantage point, prediction markets have not had a broad impact on decision making, despite all of the “anys” in the previous paragraph. Outside of political forecasting and sports gambling (and of course finance itself), I’m not aware of any groups outside of academia that invest significantly in the use of  prediction markets. Sure, there’s the Hollywood Stock Exchange that applies the fantasy sports concept to the movie industry and even startup Empire Avenue that aspires to generalize this idea even further into an “online influence stock exchange”. Still, I think it’s safe to say that prediction markets have had limited traction to date.

Many people do, however, believe that we can harness the wisdom of crowds. In particular, we as consumers rely on reviews and recommendations to inform our decisions about what to buy, read, etc. Because those decisions have financial implications for sellers, the world of online reviews has an adversarial element, where review systems face manipulation by those who would shill their own products or services. As a result, it is never clear how much we as consumers should trust the reviews we read to be sincere, let alone useful.

Which brings me back to prediction markets. Unlike most venues for soliciting collective opinion, prediction markets offer a strong incentive for accuracy. Betting on whether readers will like a book is quite different than simply offering a review that asserts an opinion without any risk to the person making the assertion. It is possible to manipulate a prediction market (e.g., by flooding it with high bets), but research suggests that such manipulations are short-lived and in fact expose the manipulator to significant financial risk when the price re-stabilizes.

So why don’t we use prediction markets instead of relying on reviews and recommendations? Perhaps we should, and it’s just a matter of time until entrepreneurs build successful businesses around this idea. But I suspect that much of the value of user-generated content today comes from contributors not thinking in market terms. While using prediction markets could solve the problem of shill reviews, it might also scare off the altruists.

Still, it seems to me that we should look for more opportunities to incent accuracy. Even altruistic reviewers have an interest in establishing their credibility, at least if that credibility determines the propagation of they opinions they share (perhaps I’m conflating altruism with egotism). The challenge may be to implement a marketplace that deals in the social currency of reputation than the hard currency of cash–while avoiding the sort of virtual currency that many people see as meaningless.

Can we obtain the benefits of market dynamics and still take advantage of the less rational motivations that drive some of the best online reviews today? I hope there are people who feel incented to work on this problem!

Some previous posts for further reading:

• Thoughts About Online Reputation
• Payola? There’s An App For That!
• Are Links A Distraction? (please let me know if you like my attempt to keep links outside the main flow)

Eric Andersen called my attention to a post by Nick Carr entitled “Experiments in delinkification“, in which Carr argues that links embedded in text are distracting, and that we’re better off treating them like the footnotes they evolved from and putting them in a block at the end of the text. It’s an interesting piece, and I see the merits of his argument. Indeed, I remember trying to read a heavily annotated edition of Nabokov’s Lolita, and it was extremely hard to maintain the flow of reading the novel while turning every few seconds to read about every last entomology reference in the text.

Nonetheless, I feel that links supply context, and I’m a fan of keeping context nearby. Indeed, I find that clicking on a link incurs a much lower cognitive cost than flipping to the back of the book, searching for an endnote. I’ve had readers specifically thank me for including links to Wikipedia entries for technical terms. I assume those readers are fully capable of finding those Wikipedia entries themselves, but that they appreciate the convenience of the links.

Some of the commenters on Carr’s post suggest that we use technology to address this tension between preserving the reader’s focus and supplying nearby context. Specifically, we can use CSS and have a JavaScript button that toggles the link style between visible and invisible. I like the idea of handing readers control of the presentation style, though I still think it’s important to pick a sensible default. At the very least, a document should be self-contained so that a reader can choose if and when to look at the material it cites. The document should also give credit where it’s due, linking to the material it cites in a way that is visible to people and search engines. Beyond that, I think it’s really a matter of author style.

Still, I’m curious what folks here–especially long-time readers–think. Do I link so heavily that it’s distracting? Would it be easier to read my posts if the links were in a block at the end? I write for you, so please let me know how I can make this blog better. I don’t have the resources to conduct cognitive load experiments, but I’m very receptive to comments.

The other day, I received a surprise package in the mail: a copy of IBM researchers David Carmel and Elad Yom-Tov‘s newly published lecture on “Estimating the Query Difficulty for Information Retrieval“. I wasn’t even aware that this book was being written, so I’m especially appreciative of the publisher’s kindness to send me a copy.

If you liked Claudia Hauff‘s recent dissertation on “Predicting the Effectiveness of Queries and Retrieval Systems” (cf. my blog post on how “Not All Queries Are Created Equal“), then you’ll love this compact lecture that review the work on pre-retrieval and post-retrieval prediction of query performance. It covers query clarity, ranking robustness, query coherence, and much more.

I’m a big fan of the Morgan & Claypool series of Synthesis Lectures on Information Concepts, Retrieval, and Services, though I’m admittedly biased. Still, I think these books are an excellent way to get an overview of a subject, and Carmel and Yom-Tov’s book delivers wonderfully. For those not lucky enough to receive free copies in the mail, I recommend Amazon, which is selling it for less than $24.

Here are the slides from Marti Hearst’s and Peter Morville‘s keynote presentations at the Enterprise Search Summit:

This morning’s Enterprise Search Summit keynote was by Peter Morville, who has written a number of best-selling books about information architecture. I’ve known Peter for a while and had the pleasure of serving as a reviewer for his latest book, Search Patterns, but had never seen him present this material live. As you can see from his slides, Peter’s presentation style is incredibly visual–almost all of his slides are screenshots or illustrations explaining his concepts. It makes for a great presentation, but a difficult text summary!

The focus of his talk, naturally, was patterns. Specifically, he advocated that we take the behavior patterns of information seekers that library and information scientists have been studying for years, and use them to inform design patterns for search user interfaces.

One point he raised that deserves a deeper dive: number of media (mobile, kiosk, TV) environments push people to browse, partly because of limitations of the medium but also taking advantage of the novelty and relative lack of user habits. Unfortunately, browsing doesn’t always scale in those environments, so search is usually available as a contingency.

Interestingly, while Peter promotes rich interfaces in many of his patterns, he noted that great results ranking plus speedy response (he uses Google “classic” as his example) does allow users to rapidly reformulate their queries while staying in the flow of the information seeking experience. He returned to Google later in his talk, noting that the new interface goes beyond ranking to support a richer user interaction.

And, like me and Marti Hearst (yesterday’s keynote), Peter advocates faceted navigation (I won’t quibble on whether to call it navigation or search) as his favorite search design pattern. He uses the NCSU library as an example not only of a great implementation but also of an organization that continues to experiment with incremental design changes. He also showed faceted search examples from other domains, including Amazon and Buzzilions.

Other patterns he discusses included question answering (his example being Wolfram Alpha) and decision making (his example being Hunch). He didn’t go deep on these, but rather invited the audience to consider a broad palette of strategies for supporting information seeking. Indeed, when I asked him about question answering, he conceded that he was a skeptic and preferred a conversational (i.e., HCIR) approach akin to a librarian’s reference interview.

His closing note was about bridging the gap between physical and digital information, where he offered a potpourri of examples (from Redbox to a tweeting plant). I work in local search, so in my case he’s preaching to the converted. But I think he’s right that everything is only recently coming together–specifically, the ubiquity of digital data on the internet and of mobile devices in the physical world that can both consume and produce that data. Many of us take these developments for granted, but it’s important that we adapt our approach to search to address what is a very recent phenomenon.

Fun stuff! I didn’t get to attend the rest of the summit, but I encourage you to check out the tweet stream at #ESS10.

The Enterprise Search Summit is taking place in New York this week, and I was lucky to be able to attend Marti Hearst’s opening keynote this morning about designing search for humans. If you’ve read her book or heard her present its material, then you’re probably familiar with the pitch she made. Still, it’s great to hear her present it live to a very non-academic audience.

Her major take-aways:

• The user’s emotional response is a key aspect of the information seeking experience.
• There is a double vocabulary problem: different ways to express same concept (cf. Furnas et al.), and users stubbornly anchor on initial query terms (cf. Kahneman, Tversky, et al.)
• Recognition is easier than recall, so interfaces need to support the recognition process.
• Don’t personalize search, socialize it!

She peppered her talk with concrete examples and scholarly references. Given that her book is available online for free, I won’t try to replicate them all here! Still, I’ll single out two Noisy Community members: FXPAL researchers Jeremy Pickens and Gene Golovchinsky (for their SIGIR 2008 work on collaborative exploratory search) and user experience designer Greg Nudelman for his proposal of faceted breadcrumbs as a search user interface.

If you missed her live, you check find a video of a tech talk she gave at Google a few months ago. You can also check out the conference tweet-stream at #ESS10.

Today I celebrate six months of working at Google. I’m having a great time, and I wanted to take a moment to share a bit about my experience thus far.

First, a bit about Google’s New York office. It is a major office–in fact, Google’s second largest office and an R&D powerhouse. New York Googlers played key roles in two of Google’s recent developments: real-time search and the results page re-design. Less visibly but not less importantly, engineers at Google New York contribute to every major aspect of Google’s technology.

My own contributions have been toward improving local search. Local search represents an increasingly large and important fraction of people’s online information seeking. At first glance, it might seem to be an easier problem than general web search, since there are only tens of millions of places, compared to tens of billions of web pages. But local search poses unique challenges–from data quality to ranking to supporting exploratory search.

Another area that I’m especially excited about is the work on structured data. The Magpie team is based in New York. You may be familiar with them as the team that developed Google Squared–which powers the “something different” links for web search.

Of course, there is far more going on at Google New York than I could hope to summarize in a blog post–including lots that I can’t talk about yet. But I hope this at least gives you a taste of what it’s like to work for the world’s best search company in the world’s best city (yes, I know I’ll take some flak for at least one of those superlatives).

If you’re interested in learning more, please don’t hesitate to reach out to me. I may have drunk the kool-aid, but I promise to be candid and as open as I can about what it’s like to go through the hiring process and what awaits you at the end of it.

Sorry for the long delay between posts. Fortunately the blogosphere has been providing ample reading material about the saga of the lost iPhone and the war of words between Apple and Adobe.

I’ve been doing some reading myself. Specifically, I just read F. Randall (“Randy”) Farmer and Bryce Glass‘s recent book on Building Web Reputation Systems. Given that I’ve been thinking a lot about online reviews and reputation systems (e.g., this recent post), I wanted to hear what the experts had to say.

In the book, Farmer and Glass categorize the motivations for user participation as altruistic, commercial, and egocentric. Commercial motives are clearly the most problematic: a review site loses credibility if commercially motivated reviews are disguised to make their commercial motives. Most review site scandals arise from this kind of deception (e.g., this one, this one, and  this one).

Sincerity is a necessary but insufficient condition for a review to be valuable to the person who reads it. There is still the “people like me” problem: sincere reviewers may still be uninformed, unreasonably biased, or may simply not share our tastes. User-generated content is an inherently subjective medium.

Given these challenges, it’s a wonder that online review sites work at all! And yet there are real success stories. My personal favorite is Amazon.com. While it has has its hiccups, Amazon nonetheless serves as a poster child for creating value by aggregating user opinions about products.

Amazon has a well-designed review policy that gets many key elements right:

• Reviewers have identities tied to purchasing history. That encourages disclosure (people use their real names) and discourages abuse.
• The reviews themselves–and even comments in discussion threads about individual reviews–are themselves reviewed as helpful or not. That may seem overly meta, but it does a lot to mitigate information overload.
• Grounding in objective information (product content, sales rank) reduces the ability to manipulate product perception through reviews.

The system isn’t perfect, but it’s good enough to be very useful.

But products aren’t the only reputable entities, to use Farmer and Glass’s term. What about service businesses, such as restaurants, gyms, etc. Or people?

If Amazon exemplifies online product reviews, then Yelp is the canonical example of a review site for service businesses. And, despite its own share of controversy, it is quite successful. But I dare say not quite as successful as Amazon. Part of the problem is that is demographics are less representative of the general online population (here’s what Quantcast says about Yelp and Amazon demographics for their US users). Also, there’s more variance in experiencing a service than in experiencing a product.

But Yelp has also has had  a credibility problem regarding which reviews they allow to be published. Perhaps the root of this problem is that Yelp’s business model depends on paid advertising from the businesses reviewed on the site, while businesses would much rather have unpaid positive reviews. In contrast, Amazon makes its money buy selling products–which at least makes it perceived to be more evenhanded.

But neither Amazon and Yelp have touched the third rail of online reputation: people. LinkedIn dabbles in this space by allowing its members to review one another, but reviewees have veto power over reviews–making the review graph more of a mutual admiration society.

A recent startup, Unvarnished, is trying to create a review site with teeth. Farmer argues on his blog that Unvarnished is breaking some major  rules:

• It displays negative karma–that is, it allows people to write negative reviews of one another and displays those reviews.
• The reviews are not clearly tied to context (e.g., were the reviewer and reviewee co-workers?).
• The anonymity of reviewers does not incent altruistic or even egocentric behavior, and is thus a recipe for abuse.

I’m not as down on Unvarnished as Farmer, but I agree it will have an uphill battle to succeed. Ironically,  for all of the public concern about Unvarnished becoming a trollfest, the reviews skew strongly positive. This is probably an artifact of how Unvarnished is growing its membership: current users ask friends to review them.

I agree most with Farmer that Unvarnished’s incentive structure seems problematic. A person’s friends will probably be inclined to write positive reviews, and may even be annoyed at having to write them anonymously. A person’s enemies may be inclined to write negative reviews as a form of attack or revenge. But it’s less clear what will incent people to write accurate reviews–or what will signal to readers that a review is trustworthy.

All in all, I think that these are early days in the online reputation space, and that there is ample room for innovation. Facebook’s recent release of “like buttons” is an ambitious attempt to boil the ocean of “social objects”. A best poster award at the recent WWW 2010 conference went to Paul Dütting, Monika Henzinger, and Ingmar Weber’s “How much is your Personal Recommendation Worth”.  Hopefully all of these attempts to research and innovate will lead to a world where we can derive real value from others’ opinions and feel incented to contribute our own.

A friend of mine at Quora invited me into their private beta a couple of weeks ago, and by now I suspect that many of you are using it–especially since I’ve somehow managed to be the top hit for [quora invite]. Speaking of which, I appreciate that those of you with spare invites have continued sharing them with the stream of folks requesting them.

Anyway, if you haven’t heard about Quora yet, here’s a summary from the site:

Quora is a continually improving collection of questions and answers created, edited, and organized by everyone who uses it.

For those of you who studied Latin, the title of this post hopefully triggers at least a faint memory of relative pronouns and declensions. It’s been suggested that “quora” is a faux-Latin plural of quorum, which in turn is the genitive plural of qui. A less arcane possibility is that quora is intended to evoke the modern-day meaning of “quorum”: a gathering of the minimum number of people of an organization to conduct business. Or perhaps “quora” is a contraction of question or answer, befitting a question-and-answer site.

How did I come up with all of these possibilities? Well, I did study Latin (semper ubi sub ubi!), but I found all of the above from the Quora entry entitled “What does Quora mean?” (membership required to view). Indeed, Quora is a great place to learn about Quora, as well as about Aardvark, Hunch, and other question-and-answer startups. Because it’s been launched as a private beta and virally marketed among friends, the community–and thus its interests–are highly skewed towards tech startups. Indeed, people seem more inclined to compare it to programmer-oriented site Stack Overflow than to Yahoo! Answers–which speaks volumes about the current user base.

All that said, is Quora a useful site? It certainly offers useful information, but that’s a pretty low bar–after all, the open web already offers lots of useful information. The better question is what Quora offers that the open web does not.

Indeed, the closed nature of the site puts it at a disadvantage to the open web: no links, search engine optimization, etc. That said, I also haven’t seen spam or any of the other abuse endemic to the open web.

In any case, I don’t see Quora as a knowledge base of first resort–except possibly to learn more about software startups. Whether by design or by virtue of its early membership, the site is a very narrow scope.

The more interesting value proposition of Quora is the community it is creating. Quora facilitates conversation, much like a members-only blog where everyone uses their real names. It’s a well-designed social site, and I like that it revolves around substantive topics.

But I worry that Quora faces a catch-22. If the focus stays narrow, I can’t imagine it creating enough utility to justify its $86M valuation. But it’s not clear that Quora can scale up to a broader scope. Given what I’ve seen of question-and-answer sites, I’m skeptical.

What Quora does have going for it is an all-star team, and I’m sure they have big plans for the site. I’m very curious to see what those plans are, and how they play out.

I hope a number of you are planning to participate in the HCIR 2010 workshop! Here is a quick update:

• Program Committee chair Rob Capra has assembled a program committee that includes Susan Dumais, Gary Marchionini, and many more.

• Folks who want to participate in the HCIR Challenge should contact me for details! You’ll have to obtain your own copy of The New York Times Annotated Corpus from the Linguistic Data Consortium (thanks to them and to The New York Times for making this data set freely available!). After that, you can roll your own application, or you can bootstrap on the Solr indexer that Tommy Chheng has graciously put together.

More details–particularly about the Challenge–are forthcoming and will be posted on the HCIR Challenge page. Meanwhile, feel free to ask me questions, either publicly here or by email, and I’ll be happy to answer them. Looking forward to seeing many of you this August!