The Noisy Channel

 

Thoughts About Online Reputation

May 2nd, 2010 · 10 Comments · General

Sorry for the long delay between posts. Fortunately the blogosphere has been providing ample reading material about the saga of the lost iPhone and the war of words between Apple and Adobe.

I’ve been doing some reading myself. Specifically, I just read F. Randall (“Randy”) Farmer and Bryce Glass‘s recent book on Building Web Reputation Systems. Given that I’ve been thinking a lot about online reviews and reputation systems (e.g., this recent post), I wanted to hear what the experts had to say.

In the book, Farmer and Glass categorize the motivations for user participation as altruistic, commercial, and egocentric. Commercial motives are clearly the most problematic: a review site loses credibility if commercially motivated reviews are disguised to make their commercial motives. Most review site scandals arise from this kind of deception (e.g., this one, this one, and  this one).

Sincerity is a necessary but insufficient condition for a review to be valuable to the person who reads it. There is still the “people like me” problem: sincere reviewers may still be uninformed, unreasonably biased, or may simply not share our tastes. User-generated content is an inherently subjective medium.

Given these challenges, it’s a wonder that online review sites work at all! And yet there are real success stories. My personal favorite is Amazon.com. While it has has its hiccups, Amazon nonetheless serves as a poster child for creating value by aggregating user opinions about products.

Amazon has a well-designed review policy that gets many key elements right:

  • Reviewers have identities tied to purchasing history. That encourages disclosure (people use their real names) and discourages abuse.
  • The reviews themselves–and even comments in discussion threads about individual reviews–are themselves reviewed as helpful or not. That may seem overly meta, but it does a lot to mitigate information overload.
  • Grounding in objective information (product content, sales rank) reduces the ability to manipulate product perception through reviews.

The system isn’t perfect, but it’s good enough to be very useful.

But products aren’t the only reputable entities, to use Farmer and Glass’s term. What about service businesses, such as restaurants, gyms, etc. Or people?

If Amazon exemplifies online product reviews, then Yelp is the canonical example of a review site for service businesses. And, despite its own share of controversy, it is quite successful. But I dare say not quite as successful as Amazon. Part of the problem is that is demographics are less representative of the general online population (here’s what Quantcast says about Yelp and Amazon demographics for their US users). Also, there’s more variance in experiencing a service than in experiencing a product.

But Yelp has also has had  a credibility problem regarding which reviews they allow to be published. Perhaps the root of this problem is that Yelp’s business model depends on paid advertising from the businesses reviewed on the site, while businesses would much rather have unpaid positive reviews. In contrast, Amazon makes its money buy selling products–which at least makes it perceived to be more evenhanded.

But neither Amazon and Yelp have touched the third rail of online reputation: people. LinkedIn dabbles in this space by allowing its members to review one another, but reviewees have veto power over reviews–making the review graph more of a mutual admiration society.

A recent startup, Unvarnished, is trying to create a review site with teeth. Farmer argues on his blog that Unvarnished is breaking some major  rules:

  • It displays negative karma–that is, it allows people to write negative reviews of one another and displays those reviews.
  • The reviews are not clearly tied to context (e.g., were the reviewer and reviewee co-workers?).
  • The anonymity of reviewers does not incent altruistic or even egocentric behavior, and is thus a recipe for abuse.

I’m not as down on Unvarnished as Farmer, but I agree it will have an uphill battle to succeed. Ironically,  for all of the public concern about Unvarnished becoming a trollfest, the reviews skew strongly positive. This is probably an artifact of how Unvarnished is growing its membership: current users ask friends to review them.

I agree most with Farmer that Unvarnished’s incentive structure seems problematic. A person’s friends will probably be inclined to write positive reviews, and may even be annoyed at having to write them anonymously. A person’s enemies may be inclined to write negative reviews as a form of attack or revenge. But it’s less clear what will incent people to write accurate reviews–or what will signal to readers that a review is trustworthy.

All in all, I think that these are early days in the online reputation space, and that there is ample room for innovation. Facebook’s recent release of “like buttons” is an ambitious attempt to boil the ocean of “social objects”. A best poster award at the recent WWW 2010 conference went to Paul Dütting, Monika Henzinger, and Ingmar Weber’s “How much is your Personal Recommendation Worth”.  Hopefully all of these attempts to research and innovate will lead to a world where we can derive real value from others’ opinions and feel incented to contribute our own.

10 responses so far ↓

  • 1 Zainul Franciscus // May 2, 2010 at 6:21 am

    I see FB “like” button as a distributed fan page system. Fans around the web can cast their “like” by clicking an FB “like” button. Let just say we have 5 “likes” on a “Beginner Guide to Python Programming” book. We do not have enough information to judge how relevant this “likes” to our need. We do not have information on the background of these people who like this book. We do not know why they like the book. Nevertheless FB “like” button has created more meaning towards our gesture on the web. It is one step to create a more meaningful semantic on the web.

  • 2 Nitya // May 3, 2010 at 11:28 am

    First of all — thanks for highlighting a good book for my to-read pile!

    This was a great post and fodder for a longer discussion but two quick thoughts come to mind. Sorry for the long comment (I am waiting on a testing run to complete so I had time)

    1.
    Reputation should be some combination of reliability (proof that reviewer has the expertise to give the review), recency (how fresh is that reviewer’s knowledge of the reviewed content) and rating (the actual review content).
    I think the recency attribute is not sufficiently exploited by many systems. The related side effect of frequency (i.e., did you rate item in isolation or as part of a cluster) is equally interesting. As the Netflix solution(http://pragmatictheory.googlepages.com/NetflixPragmaticTheoryV6.pdf) showed, user’s opinions do change with time — particularly as they acquire more knowledge on the topic and more items to correlate (balance) reviews across. Users may be harsher or more pragmatic in reviewing items later than they would be when caught up in the emotion of the moment.
    It would be interesting to see systems instrumented to allow users to periodically re-evaluate or adjust ratings and study how this changes overall reviews. For instance, when I review a book as 4 stars on Amazon, if it then popped up samples of books that I had rated as 3, 4 and 5 stars for comparison – would I change my rating? Or would I alter a previous rating?

    2.
    The second comment was regarding Facebook. Right now their “Like” button simply allows you to state your opinion without necessarily targeting it at any user — however, it’s a short step to targeted likes where the user, in addition to clicking the Like button, can now select from a pop-up of his social graph, specific friends to send that recommendation to. More user effort but invariably cleaner (more valuable) intent data. We’re already seeing some behaviors like this on Twitter where folks “RT” a link but then add a “/cc” at the end (with a list of other twitter IDs) to indicate they want this brought to the attention of those specific users.

  • 3 Daniel Tunkelang // May 4, 2010 at 11:22 am

    Zainul (and Nitya): I agree that the “like” button is a pretty minimal signal. If anything, the more interest potential consquence of FB rolling it out is that more site authors with adopt RDFa.

    Nitya: Great point about recency. Stale reviews are a real problem for service businesses, and Farmer and Glass discuss strategies for aging out reviews. But I also like your point about how time gives us (or changes our) perspective. Makes me think about Kahneman‘s research on experienced vs. remembered utility.

  • 4 Daniel Tunkelang // May 4, 2010 at 3:50 pm

    Also, law professor Eric Goldman emailed me to point me to a recent presentation he made on regulating reputational systems.

  • 5 F. Randall Farmer // May 5, 2010 at 10:39 am

    Great post! I’m glad you liked the book.
    For your readers – the draft text is available in wiki form at http://buildingreputation.com/doku.php

    I agree that Amazon has is the bellwether of online product reviews – and on that note would like to request: Could you write a brief review of Building Web Reputation Systems for us? :-)

    Link provided for your convenience:
    http://www.amazon.com/Building-Reputation-Systems-Farmer-Randall/dp/059615979X

  • 6 Barry Hurd // May 5, 2010 at 8:15 pm

    I agree that most review sites have left the personal level unaddressed, but in a more holistic viewpoint has Google (aka search) not created a reputational layer?

    Unvarnished has some validity in attracting statements of concern, but this is not a reputational system…. it is a search engine optimization tactic.

    Unvarnished is simply “looking for monkeys” to create sources of user generated content that will SEO properly and rank for individual names.

    As a business model Unvarnished may work, but as a reputational model it is simply abusive in nature.

  • 7 Daniel Tunkelang // May 7, 2010 at 12:29 am

    Randy: you’re welcome. And the review should be up on Amazon shortly! Which reminds me: I’d love to see more reviews of my own book

    Barry: web search engines certainly treats web pages as reputable entities. And SEOs represent the contributors with commercial interests.

    As for Unvarnished’s goals, I’ll take Peter at his word. But, even if he has the best of intentions, I think he has an uphill battle to succeed.

  • 8 Why Can’t We Just Use Prediction Markets? // Jun 9, 2010 at 8:11 am

    […] Thoughts About Online Reputation […]

  • 9 Quo Vadis, Quora? // Jan 9, 2011 at 6:27 pm

    […] raises lots of interesting questions about search, user experience, knowledge management, and online reputation. How could I […]

  • 10 Loci 2010 – Ted Paff | Understanding Google Maps & Local Search // Jan 13, 2011 at 6:02 am

    […] found this article and this article to be thoughtful commentaries on review […]

Clicky Web Analytics