Yes, that’s a provocative title. But check out this far more provocative statement by Emil Protalinski in an Ars Technica article entitled “Why Microsoft continues with search: it’s still not solved“:
What Microsoft is saying here is that everyone should be able to find what they are looking for on their first attempt, every time.
I hope that’s not what Microsoft Live Search director Stefan Weitz, whom Protalinski interviewed, said or meant. Even Google, to the best of my knowledge, has never made so bold a claim. Not all information needs are amenable to one-shot queries, even using a divinely inspired search or “answer” engine.
I’m glad to see that Microsoft is taking search seriously, and I hope that their latest Kumo efforts create more credible competition for Google. But let’s not chase delusions.
For more details, check out my presentation on reconsidering relevance.
The Noisy Channel 
11 replies on “Why Are People So Clueless About Search?”
Everyone should be able to find what they want on their first try every time, or at least 95%. It just isn’t possible yet. Without that goal, why keep improving the search tools?
LikeLike
I recommend you take a look at the presentation I cited. In a nutshell, I think search tools should aspire get near-100% accuracy on the simple information needs that allow for it. But there’s a broad class of information needs for which users aren’t even sure of what they want on the first try. Even when the user is sure, then is often a loss in translation between a user’s information need and an expressed query.
The job of search engines is to help users figure out what they want and ultimately find it. Focusing on the one-shot case can only take us so far.
LikeLike
I find it interesting that multitude of dimensions we originally saw to the search engine problem (I mean ‘we’ loosly, I wasn’t around back in the 50’s) are multiplying as the technology matures. It’s like the ever expanding universe.
To start with the idea was to retrieved the correct document by using a multitude of techniques that were text based, be it NLP, classifiying and so on.
Now we that we have had a lot of experience testing these systems, with billions of queries a day and that as data increases (not only in size but number of different structures…), we see that part of the problem is actually the user.
From tests I have been running on user queries both in natural language systems and on keyword queries, I can say that it appears that people compromise with the results they have been served and the information they were originally after.
Also you can see evidence of them not really knowing what they want but knowing they have found it when they stumble upon it. There are many other examples.
“The job of search engines is to help users figure out what they want and ultimately find it. ”
Absolutely. I think that query intent is a very valuable and interesting research area right now.
LikeLike
The generous interpretation of “everyone should be able to find what they are looking for on their first attempt” is that the initial interaction with the interface should provide useful options for further interaction that (eventually, if not immediately) lead to the view of the information needed by the user. The most concerning slice of users mentioned in the Weitz interview are the quarter that abandon their search altogether. Some of this is unavoidable – some users want results that simply can’t be had (even social search won’t tell me whether P=NP), or are simply not capable of interacting with the system (arguably no matter how good we make the interface, some people still won’t get it). But 25% attrition!? There’s no doubt we can do better than that. The goal that “everyone should get useful feedback towards what they are looking for on their first attempt” seems like a useful aspiration towards addressing this problem, and a reasonable (if generous) interpretation of the idea behind the original quote.
LikeLike
I agree. Users are actually patient with a process (up to a point) if it rewards them for their investment–that’s really Gary Marchionini’s point in “Toward Human-Computer Information Retrieval“. Attrition is a bad sign–that should only happen when users actually confidently (and correctly!) terminate their search process because the information they seek is actually unavailable. I discuss this case in my post on “Precision and Recall“.
LikeLike
Depending on how users’ intent is inferred, some of the 25% “abandoned” searches may in fact have shown people the answer the want in the snippet or allowed them to copy a URL, or use the displayed results directly in some way without interacting with any of the links that represent retrieved documents.
LikeLike
I’m not sure how valid “abandonment” is, as a measure of search quality, anymore. I’m certainly part of the 25%, whether I find the information I am looking for in the snippet, or I copy the URL because want to go to the site’s root (local site search often returns fresher content, in my experience).
I know that in my case, a better measure of search success would be how many iterations of refining the query string I have to go through, before I find what I am looking for.
LikeLike
First, let me say how nice it is to see thoughtful discussion in comments.
Second, let me echo Daniel’s first comment. I didn’t mean to imply that search engines should always return a single answer or even that an engine should ‘get it right’ on the first try. As we know, searches are often classed into three major categories of queries: navigational, transactional, and informational. The nav queries should generally be short, to-the-point, and as frictionless as possible as there is generally an ‘answer’ to your question that is more accurate or relevant than others. The informational and transactional (esp the informational) often tend to be more exploratory and today require multiple queries and refinements as people attempt to gather enough info to make a decision. Informational queries tend to be lengthy and complex and we find that people often come back to those queries over the course of time (think days or weeks even). The challenge of making informational and transactional classes of queries return greater “value per minute spent searching” is one we’re really looking at.
So no, I don’t believe that an engine will become prescient enough to return a single, one-query-and-out answer for all queries – primarily because some of the hardest “questions” often can’t be addressed with such an experience. Engines should get out of the way when they can and guide searching to help people make decisions when they need it.
LikeLike
Stefan, it’s a pleasure and a privilege to have you join this discussion. And I am gratified to know that we’re on the same page.
Indeed, I see informational searches as an area where Google often falls short, and hence one where Microsoft and others would do well to focus efforts. I’m glad to see that you and your colleagues are thinking about this challenge, and looking forward to the solutions you bring to bear on it!
Any thoughts on how to evaluate “value per minute spent searching”?
LikeLike
We are looking at a number of ways to think about value/search. One is a time-to-task metric (i.e. how long does it take someone to complete foo task and how do we reduce that). As we see searches really becoming chains of related queries, this seems like an interesting measure. I’m also looking at things like reduction in repetitive searches (altho that would still likely manifest in a time-to-task metric) and how quickly a user can reach a ‘satisfied click’ with algo results (whether task-based or not).
Beyond that, I haven’t given it tons of thought. But I do like the concept around how we increase the ROI of searchers’ activity.
LikeLike
Stefan, have you looked at any of the “games with a purpose” work that Luis Von Ahn is leading at CMU? I’m particularly intrigued at what sort of task-driven evaluation of interactive / exploratory information retrieval systems could be done using games like Phetch. I also wrote about this approach in a position paper for the Information Seeking Support Systems workshop, which I posted here.
LikeLike