Categories
General

Why Can’t We Just Use Prediction Markets?

Prediction markets were all the rage a few years ago, two of the most notable being the Iowa Electronic Market forecasting electoral results and the now defunct Tradesports offering a similar platform for betting on sports events. There was even a proposal to have the US government run a prediction market for terrorist attacks.

In a prediction market, any event with a quantifiable (e.g., binary) outcome can be converted into an asset. At any given time, the asset value corresponds to the market prediction of the probability of the outcome. Just as in any security market, participants determine the value through their buying and selling actions. In principle, this framework allows any event with a quantifiable outcome to be predicted by a marketplace.

But, at least from my vantage point, prediction markets have not had a broad impact on decision making, despite all of the “anys” in the previous paragraph. Outside of political forecasting and sports gambling (and of course finance itself), I’m not aware of any groups outside of academia that invest significantly in the use of  prediction markets. Sure, there’s the Hollywood Stock Exchange that applies the fantasy sports concept to the movie industry and even startup Empire Avenue that aspires to generalize this idea even further into an “online influence stock exchange”. Still, I think it’s safe to say that prediction markets have had limited traction to date.

Many people do, however, believe that we can harness the wisdom of crowds. In particular, we as consumers rely on reviews and recommendations to inform our decisions about what to buy, read, etc. Because those decisions have financial implications for sellers, the world of online reviews has an adversarial element, where review systems face manipulation by those who would shill their own products or services. As a result, it is never clear how much we as consumers should trust the reviews we read to be sincere, let alone useful.

Which brings me back to prediction markets. Unlike most venues for soliciting collective opinion, prediction markets offer a strong incentive for accuracy. Betting on whether readers will like a book is quite different than simply offering a review that asserts an opinion without any risk to the person making the assertion. It is possible to manipulate a prediction market (e.g., by flooding it with high bets), but research suggests that such manipulations are short-lived and in fact expose the manipulator to significant financial risk when the price re-stabilizes.

So why don’t we use prediction markets instead of relying on reviews and recommendations? Perhaps we should, and it’s just a matter of time until entrepreneurs build successful businesses around this idea. But I suspect that much of the value of user-generated content today comes from contributors not thinking in market terms. While using prediction markets could solve the problem of shill reviews, it might also scare off the altruists.

Still, it seems to me that we should look for more opportunities to incent accuracy. Even altruistic reviewers have an interest in establishing their credibility, at least if that credibility determines the propagation of they opinions they share (perhaps I’m conflating altruism with egotism). The challenge may be to implement a marketplace that deals in the social currency of reputation than the hard currency of cash–while avoiding the sort of virtual currency that many people see as meaningless.

Can we obtain the benefits of market dynamics and still take advantage of the less rational motivations that drive some of the best online reviews today? I hope there are people who feel incented to work on this problem!

Some previous posts for further reading:

By Daniel Tunkelang

High-Class Consultant.

23 replies on “Why Can’t We Just Use Prediction Markets?”

Indeed, “futarchy” was named by the New York Times as a buzzword of 2008 — that only reinforces my feeling that interest in prediction markets peaked two years ago.

But I think you (and Felix Salmon) may really be hitting it on the nail. We may want to share our opinions, but we’re not interested in being judged. Nonetheless, many review sites allow reviews to be rated–and review helpfulness seems to be an extremely successful feature. Or is that something else entirely? I suppose reviewers may always think they are right, and that subjective helpfulness scores aren’t the same as being proven wrong by incontrovertible facts.

Like

By Post Author

I agree with your assessment that the hotness of prediction markets peaked 2 years ago. I used to use Predictify and earned about $8 before they went belly up. I was also a member of a reading group at CMU on prediction markets. Not sure that led to anything (it was around the time I graduated), which was … 2 years ago.

Like

A prediction market requires enough folks to care to make the market statistically useful.

Many things people review may only get a handful of reviews. (I might even say most…)

Sure prediction markets work well for big movies, top 40 music, American Idol contestants, major stocks, major books and so on.

I think we have two different things to solve.
a) how does something (anything) get discovered enough to make it relevant to enough people that

b) a prediction market becomes useful to separate winners and losers, recommendations, etc (i mean this in an abstract sense)

One last thought. Outcome is very hard to discretely define more a good deal of things in life. Which you hint at with your second paragraph.

For instance I enjoy books that are a “good story”. That’s really hard to define. I have found good stories in books no one else likes much less buys. In fact, if you asked me what a good story is today and then tomorrow, I’d probably give you to different answers and show you two different example books.

All the flaws and inefficiencies of reviews are kind of their useful parts. It’s not so much their ability to predict as it is the color they provide. e.g. sometimes I watch a movie or read a book because it tells me something about the person who recommended it, not because I expect to love the book. A further confusion of defining outcome…

Nice post. Thanks for making me think this early!

Like

How come no one mentioned Intrade.com? It is still going strong and has futures in many areas such as sports, politics, and entertainment. I often check it to gauge how people think about the probability of certain events.

Like

Perhaps I’m just being impatient–though I’m not quite ready to treat the Gartner hype cycle as gospel.

As for Intrade, it was an offshoot of Tradesports, and I apologize for not mentioning it. But I believe Intrade also peaked in 2008, even if it has managed to survive its parent.

Like

By Post Author

I’m a little biased about this because I’m the founder of a company that makes prediction market software but I would agree that even though there are actually dozens of companies running prediction markets internally, their impact on decision making varies widely. The main issue we’ve seen is a cultural one. Whether it’s from data or 1,000 people in their company that have predicted with a high probability something is going to happen, decision makers still tend to rely on their own gut or on the advice of a few confidantes. It’s actually kind of an interesting design problem – how do you get executives to trust the input of their employees; is there a way that information can be communicated differently to help bridge that gap? Anyways, there ARE other benefits (in a business setting) besides just the predictive aspects: awareness of issues and risks increase among participants, the availability of an anonymous venue to express what you really think vs. your words/actions being influenced by office politics, breaking down organizational boundaries, etc.

The hype about prediction markets – public ones anyway, probably goes hand in hand with US elections, since that is when they get written about the most in comparison to polls as journalists look for other signals about what may be going on. On the business side however, they are slowly gaining legitimacy as a viable forecasting and risk management tool and some of the bigger consulting firms and tech providers are starting to give them more attention, but just like anything else in a corporate setting there is a long slog to get past the cultural challenges. Afterall somewhat uniquely to prediction markets, they are designed to make information more transparent and the decision making process potentially more democratic which can be a little scary for certain people.

Like

Many people do, however, believe that we can harness the wisdom of crowds. In particular, we as consumers rely on reviews and recommendations to inform our decisions about what to buy, read, etc.

Perhaps I am going out on a limb here and exposing my lack of understanding about the whole “wisdom of crowds” concept, but I see a huge conceptual difference between Amazon reviews and prediction markets. The latter is indeed “wisdom of crowd”. The former is not.

Why are Amazon reviews not “wisdom of crowd”? Because they are technically not “aggregations”. See Surowiecki’s four conditions that need to be met to have crowded wisdom:

http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds#Four_elements_required_to_form_a_wise_crowd

See the fourth point. Aggregation. “Some mechanism exists for turning private judgments into a collective decision.”

An Amazon review is not a collective decision. It is a solitary author (or a small group of solitary authors), with his or her own individual, opinionated, editorial voice. That voice hasn’t been “aggregated” into a collective decision. It is presented to the reader…raw.

One of the problems is that people see the usefulness of Amazon reviews, attribute that to “wisdom of crowds” and then believe that the same level of usefulness is achievable in other arenas (sports betting, elections, etc.) So in my (possibly incorrect) opinion, the main problem is that Amazon reviews are NOT wisdom of crowds.

Perhaps that’s why prediction markets are not catching on. The successes that are trying to be emulated (Amazon reviews) are working precisely because those successes are not actually “wisdom of crowd”. When true “wisdom of crowd” (aggregation) is applied, it then wrecks the very thing that made Amazon work in the first place.

Don’t know for sure. But it’s a reasonable hypothesis, eh?

Like

Jeremy, great to hear from you!

I don’t know exactly what Surowiecki means by aggregation, but I would think that Amazon’s (average) product ratings and collaborative filtering both count as aggregating collections of private judgements. The individual reviews are raw, but I imagine that many consumers focus on the aggregate statistics.

But my concern is whether reviewers are accountable for the sincerity and utility of their reviews. To some extent they are: other users tag reviews as helpful. I don’t think that is quite as clean as the prediction market model, but I do believe it improves the overall quality of reviews.

But perhaps you have a different concern in mind, which is that we have to be careful not to confuse prediction of objective outcomes with the sharing of truly subjective opinions. Reading raw reviews may simply be a great way for a consumer to learn about a diversity of opinions. Still, I see a place for market mechanisms to incent sincerity and even utility on the part of reviewers.

Like

By Post Author

Average product rating is Surowiecki-flavored aggregation, yes. As is CF. If that’s all we’re talking about, then yes, WOC works.

But I’ve always found the raw reviews to be much more useful than the average rating score. I read the 5’s and I read the 1’s. If that doesn’t give me enough info, I tiptoe into the 4’s and the 2’s, too. That’s usually enough to give me a sense of not only the positives, but the potential drawbacks.

But there is one other issue here, too. And that is Amazon is concerned with WOC over events that have already “happened”. Someone already recorded that album, or manufactured that camera, in order for users to have acted on that information and purchased the item.

In prediction markets, the things that are being tossed around don’t even exist yet. And conceptually, that has a much different flavor to me than Amazon.

Is that what you’re getting at, with the objective outcome vs. subjective opinion distinction? If so, then I think I agree.

Like

I agree that sincere raw reviews are useful–and often convey far more information than terse numerical scores. Though it’s also possible to distill pros and cons (e.g., Buzzilions.com and Pluribo).

As for learning from the past vs. predicting the future, isn’t the point of a review to accurately predict whether you’ll like a product? I think the real issue here is that these predictions are focused on individual behavior, while prediction markets tend to have a broader scope. Though predicting whether a review will be helpful starts to broaden the scope a bit. Indeed, a reviewer might actually be willing to bet on the helpfulness of his or her reviews–or on those of other reviewers–and thus earn credibility.

Maybe this is just crazy talk. But I do think that we could improve the state of online reputation by creating market frameworks for it.

Like

By Post Author

isn’t the point of a review to accurately predict whether you’ll like a product

But the “product” has already happened. It’s already in the past, even if you personally haven’t bought it yet. The product may or may not be in your future, but objectively, it’s not on the future. It’s in the past.

I think that’s what you’re getting at w/ the distinction between individual behavior vs. broader scope?

Like

By the way, this is the same distinction that I am always trying to get people to see in the “social search” versus “collaborative search” domains.

In social search, you’re passing around “products” (e.g. existing search results) that others have already found. The results already exist, and you’re just using your social connections to propagate them.

In collaborative search, on the other hand, you’re working with a small team (2-3 people) of people, and NONE of you have found the necessarily results yet. There is no existing information to be propagated, because it’s all “future”, if you will.

And to me one of the biggest issues with prediction markets is that, when you’re talking about products that don’t exist yet, because they’re in the future, is it better to have a hundred thousand random people do the prediction? Or is it better to have a small team work together to find as much information as possible, to make the prediction?

Hmm.. I think this deserves a full blogpost. Perhaps Gene and I will write something on the FXPAL blog for this coming Monday.

Like

Jeremy: the product has happened, but future purchases haven’t. But I suppose that predicting purchasing behavior or consumer opinions maybe be quite different from predicting the sort of event outcomes typically associated with prediction markets.

Annie: thanks for the link. Perhaps prediction markets are overrated even in their best domains. But that still doesn’t convince me that they wouldn’t be an improvement over trusting collective opinions that aren’t subject to market dynamics–especially in cases where some of those expressing opinions might have incentives to manipulate the results.

Like

By Post Author

Jeremy: the product has happened, but future purchases haven’t.

Future purchases haven’t happened, but at least one event of the exact same type — a purchase of that product — has. Someone, somewhere, has purchased that product. So the abstract event, the “purchase of the product”, has indeed already occurred, and is already in the past.

But I suppose that predicting purchasing behavior or consumer opinions maybe be quite different from predicting the sort of event outcomes typically associated with prediction markets.

Yes, exactly. Like the “terrorism” prediction markets that were popular a few years ago. The gov’t wanted to figure out when the next attack would occur, and what that attack was going to be, using prediction markets. And the problem is that event hasn’t even been “purchased” once. It’s not just that you personally haven’t “purchased” that terror attack “product” (i.e. not just that you personally haven’t been attacked by that event), but that no one has.

With product purchases, the same event keeps happening again and again and again, just by different people. With terror prediction markets, or superbowl prediction markets or whatever, the event hasn’t even occurred once. To anyone. It’s not like some of us have seen the outcome of the next superbowl, and some of us haven’t seen it because we haven’t yet decided to purchase the pay-per-view option of the game. It’s that no one has seen it.

And again, not to go too deeply into it, I see this same distinction in social search vs. collaborative search. Propagation of existing events to new people (social) vs. creation of new, hitherto non-existent information (collaborative).

Like

@ jeremy
Very informative dissection – I learnt something new!

Btw, curious to know if collaborative search is a market need?

@ russ foltz-smith
Wrt “good story” books. Isn’t the query problem, given one or more books as examples (of a ‘good story’ in this case) find other similar books?

Like

Interesting discussion, but profoundly wrong. First of all, ”wisdom of the crowds” has very little to do with prediction since the more data we get from the hive, the more normal distributed it gets. Just read any simple text on Central Limit Theorem and you get what I mean, …because if you had read it already and understood it, this discussion would look very different. An example. In predictive medicine, only extremely simple additive or multiplicative models are in clinical use (one example is SOFA scoring). This corresponds to the “stupidity of the masses”-model and is only used as a very simple guide. For more complex predictive medical solutions, the interest is in anomalies and highly epistatic relations between variables. Throwing the “crowd” at this kind of complexity would just destroy everything interesting, smoothed by normal distributions. The more we understand about systemic interactions in social networks the clearer we see how meaningless the interactions are. There are a few exceptions, creation evolutionary “saltation” (term from biology) patterns, but most of it the social interactions creates huge amounts of meaningless noise. The same pattern can be observed in nature with low level organisms such as virus or bacteria, so the sustaining power is there but completely brain dead. My second point is much simpler. Social models for predictions is something that has been tried over and over. Both companies and very strong academic research teams. Outcome? Very little compared with other predictive technologies around, reflecting the things I’ve said above. Now I’m off to Facebook,… must add my knowledge to the hive… 😉

Like

Harry, I suspect a lot of people here are quite familiar with the Central Limit Theorem. But I don’t follow your argument. You seem to be assuming that people are voting sincerely, and that the main concern is how to combine evidence (e.g., in a way better than uniformly sampling the population and taking the mean). But that’s not the concern I am trying to address. Rather, I see prediction markets as a way to help avoid people casting shill votes. That may not work either, but not for the reasons you describe.

Like

By Post Author

Comments are closed.