The Noisy Channel

 

I’d Like To Have An Argument Please

May 30th, 2011 · 9 Comments · General

If you Google [relevance theory], you’ll discover this Wikipedia entry about a theory proposed by Dan Sperber and Deirdre Wilson arguing that, in any given communication situation, the listener will stop processing as soon as he or she has found meaning that fits his or her expectation of relevance. The Wikipedia entry offers the following example of this principle:

Mary: Would you like to come for a run?

Bill: I’m resting today.

We understand from this example that Bill does not want to go for a run. But that is not what he said. He only said enough for Mary to add the context-mediated information: i.e. someone who is resting doesn’t usually go for a run. The implication is that Bill doesn’t want to go for a run today.

This theory may call to mind the Gricean Maxims — indeed, Sperber and Wilson borrow heavily from Grice’s work.

But I mainly bring up relevance theory to introduce Sperber to those unfamiliar with him. My friend (and Endeca co-founder) Pete Bell recently called to my intention an article by neuroscientist Jonah Lehrer entitled “The Reason We Reason“. The article reviews the “hot hand” fallacy and then proceeds to cite a new theory by Sperber and Hugo Mercier:

Reasoning is generally seen as a mean to improve knowledge and make better decisions. Much evidence, however, shows that reasoning often leads to epistemic distortions and poor decisions. This suggests rethinking the function of reasoning. Our hypothesis is that the function of reasoning is argumentative. It is to devise and evaluate arguments intended to persuade.

The full article by Mercier and Sperber runs over 17K works and is entitled “Why do humans reason? Arguments for an argumentative theory“.

As someone who has spent most of his professional life thinking about information retrieval in practical contexts, I automatically relate relevance theory to relevance in the context of information retrieval. Relevance has been a subject of intense debate in the information science community (Tefko Saracevic tells the story wonderfully). Indeed, a key reason that I created the HCIR workshop was the belief that information retrieval researchers and practitioners (i.e., search engine developers) were placing too much emphasis on an objective notion of topical relevance, and not enough focus on the user.

Mercier and Sperber’s theory offers an interesting challenge to information retrieval researchers: perhaps a user’s information need is less about arriving at the truth and more about finding confirmatory evidence to support a preconceived conclusion. If so, should we adjust our notions of relevance accordingly? Also, if we evaluate or inform search quality based on observed user behavior (such as click-through behavior), then are we already inadvertently conflating topical relevance with users’ confirmatory bias?

Many people have noted that personalization gives us the truth we want: recent examples include Robin Sloan and Matt Thompson’s EPIC 2014 and Eli Pariser’s The Filter Bubble. Despite the consensus that over-fitting information access to our personal tastes is a bad thing (perhaps even dystopian), technology seems to relentlessly push us in this direction. Moreover, some degree of personalization is clearly useful — such as prioritizing information that relates to our personal and professional interests.

Nonetheless, anyone working in the area of information seeking systems should be concerned with the question of the user’s goal in using that system. Many of us take for granted that the user’s main goal is truth seeking, and we design our systems accordingly. What can or should we do differently if the user’s main goal is not informative but persuasive? Is the user looking for an answer…or an argument?

9 responses so far ↓

  • 1 John McCubbin // May 31, 2011 at 4:26 am

    You suggest that the goal may be more about finding “confirmatory evidence to support a preconceived conclusion” by seeking “persuasive” rather than “informative” results.

    This reminds me of Personal Construct Psychology as developed by George Kelly (1955). Kelly’s fundamental postulate was the “A person’s processes are psychologically channelized by the way in which he anticipates events”. He saw the individual as a “personal scientist” always trying to confirm his predictions and validate his personal constructs of the world. Thank you for the thought provoking post, time to re-visit Kelly and PCP.

  • 2 Daniel Tunkelang // May 31, 2011 at 7:18 am

    John, thanks for the comment! I see the connection to Kelly’s PCP, but Kelly’s theory strikes me as having more faith in human rationality. Mercier and Sperber are essentially saying that confirmation bias (which is a bane of the scientific method) is a feature, not a bug, in our reasoning process.

  • 3 AJ Kohn // Jun 1, 2011 at 9:09 am

    The idea that relevance does not equal truth, but may instead is determined by bias or subjectivity helps inform the debate around search quality.

    I believe many conflate relevance and quality. The two are clearly related and entwined but I don’t see them as synonymous. Your supposition that relevance is itself subjective in nature takes that a bit further.

    The confirmation portion resonates with my experience with reviews. There are some who search for reviews after a purchase as a way to pacify buyer’s remorse.

    Anecdotally, I also believe that much of the research around reviews is confirmation based. How many times does a person determine that they want to buy a certain product and then use research and reviews to confirm that they’re making the right decision? How often do users truly change purchase intent based on that research? I’d hazard to guess that number is relatively low.

    Finally, this reminds me of a study by Jakob Nielsen.

    http://www.useit.com/alertbox/search-skills.html

    “users change search strategy only 1% of the time; 99% of the time they plod along a single unwavering path”

    Nielsen was looking to confirm different theories, but I think it may also shed light on the idea that we’re seeking a confirmation.

    Many thanks for the thought-provoking post.

  • 4 Daniel Tunkelang // Jun 1, 2011 at 11:07 am

    AJ, thanks for the comment and link. When I saw Nielsen’s study, I read it as suggesting that users are at worst uninformed and at best satisficing. It hadn’t occurred to me that users might simply be looking for confirmatory evidence.

    As Alice said, curiouser and curiouser!

  • 5 Michael Martinez // Jun 1, 2011 at 11:11 pm

    If I want confirmatory evidence I’ll search for sites that agree with my views by expressing those views.

    If, however, I want to discover something new, I’ll search for sites that don’t necessarily have anything relevant to my views. In that case, only the irrelevant is relevant. For example, if I want the best price on a new computer, what I think is a good price isn’t relevant to what is available. On the other hand, if I want to confirm that the best price I can get is X, it will serve my interest better to find a price that is X – Y.

    Sometimes, even the desire to confirm a point of view is only a disguised attempt at discovery.

    You cannot base a sophisticated information retrieval theory on a simple assumption.

  • 6 Daniel Tunkelang // Jun 2, 2011 at 8:20 am

    Michael, it sounds like you’ve done more than most to overcome the human tendency toward confirmation bias. If so, good for you! And good for all of us who manage to do it in varying degrees.

    But the idea that information seekers have a primary goal of discovering the truth is just as much a theory as the idea that they are seeking to confirm a point of view. Perhaps status quo bias favors it. But it doesn’t make the theory any more “sophisticated” than the argumentative theory that would be a corollary of Mercier and Sperber’s argumentative theory of reasoning.

  • 7 Michael Martinez // Jun 2, 2011 at 11:44 am

    I’ve been studying search for a few years from the commercial side. I don’t think IR science is ready to produce the Universal Searcher Motivation Theory just yet.

    People are treating Google like a personal assistant more than anything else, in my opinion — especially in the U.S., perhaps less so overseas. That’s a very complex relationship, in my opinion.

  • 8 Foo for Thought // Jun 29, 2011 at 5:02 pm

    […] actually learn from data or do we engage in assault by data to defend preconceived positions (cf. argumentative theory). Like all of the conference, the discussion was under “frieNDA”. so I’m being […]

  • 9 Science as a Strategy // May 2, 2012 at 7:41 pm

    […] on understanding human irrationality. I also threw in Mercier and Sperber’s recent work on reasoning as argumentative. The summary: don’t trust anyone’s theories, not even […]

Clicky Web Analytics