Guest Post: Diego Basch on The Need for Speed

Diego Basch is the CEO and founder of IndexTank, a hosted search service that powers major web sites such as Reddit, Twitvid, blip.tv, as well as providing a WordPress plug-in for blogs (like this one). Diego gained his search experience working with Inktomi, where he wrote some of the world’s first web-scale link analysis algorithms. He is on a mission to make every search box blazing fast and useful.

So much brainpower is spent solving the wrong problems. The world is filled with solutions looking for problems that nobody has — as illustrated by a Google query for [stupidest inventions ever]. More often, people focus narrowly on a particular approach when they should focus on the problem the approach is intended to solve. Or they take a solution for one problem and assume it will apply to another.

Consider the emphasis that search engine developers place on relevance ranking. It is not hard to understand why web-scale search engines emphasize relevance. For example, a search on Google for [emergency locksmith] returns tens of billions of web pages, among which there are only a handful results that you want. Google must filter out the growing number of lead generation companies that spend a ton of money trying to game its results.

Most web and application developers are familiar with the concept of relevance, so they naturally assume that it should be the primary concern when they add search to their own sites or apps. When I talk to people who want full-text search for their 40,000 book titles or 100k classified ads, they ask me about all the ways they can tune relevance. But often they are focusing on a solution, rather than their fundamental problem.

Developers are (or should be!) trying to improve the user experience of their application search. Too often they wrongly assume that relevance is the single most important factor for optimizing this user experience. Let’s surface this confusion in a concrete example.

As a rock climber, once in a while I feel the aches and pains caused by the sport. As the years go by it’s very important to keep your tendons healthy if you do not want to take forced breaks (or type with one hand!). Rockclimbing.com is one of the most popular climbing sites, and I know some medical professionals who occasionally answer health-related questions there. Let’s search there for [tendon injury prevention].

In the above example, part of the problem is that the search results do not have contextual snippets. Maybe there is relevant information hiding behind a click, but the user has no way of knowing. More generally, there’s no hint as to what results could be better. Information such as score of the answer (which is available), the author’s bio (e.g. “climber, physical therapist”) would make the decision easier. If you need to click and scroll, search within the page, go back and try something else, then the search engine is wasting your time. 

Which brings us the broader point: when users search, they want to spend the least amount of time possible getting to the information they want. Relevance is a means to this end. In particular, clicks and typing costs users time. That time can come from page load, rendering, repeated use of the back button, and of course typing (and re-typing) search queries.

Some application search engines really nail the user experience. Let’s say we’re looking for the movie Koyaanits-however-you-spell-it. Go to the Internet Movie Database (IMDB) and start typing k-o-y-e — and there it is, as the second result. Notice that there is a ton of irrelevant stuff around it but it doesn’t matter. I see what I want very quickly.

Hopefully these two examples serve to illustrate the broader point: search engines should not focus on relevance as an end in itself, but rather on whatever helps users find the information they want as quickly as possible. That means offering contextual snippets, instant feedback, and of course snappy response times. Give users speed, and you will make them happy.