Comments on: Google’s Chief Economist Hal Varian Talks Stats 101

By: Daniel Tunkelang

Daniel Tunkelang — Sat, 22 Aug 2009 17:00:53 +0000

Panos, I think we’re on the same page. That’s what I meant by saying that the ability to run twice as many simultaneous tests without noticeably disrupting overall user experience is a major competitive advantage. But, as you point out, that’s a somewhat inconvenient truth if you’re trying to argue that scale isn’t that big an advantage.

BTW, I don’t think this is about Google playing down the the Microsoft-Yahoo agreement. Rather, it’s to dismiss the argument that Google has a monopolistic advantage because of its scale. I.e., he’s arguing that Microsoft doesn’t need Google’s scale to be competitive.

By: Panos Ipeirotis

Panos Ipeirotis — Sat, 22 Aug 2009 16:48:06 +0000

What Varian is saying here is that for tuning you do not need to run the experiment on all users. Correct, if you tune elements on a wide-scale (e.g., the infamous “what shade of blue to use for our interface element”)

However, there are many aspects of tuning in which you do run into sparse data problems. How do you estimate, say, clickthrough for queries that are typed only a few hundred times a day? Or how do you estimate clickthrough for an ad-query pair? In such cases, you *do* have to deal with sparse data, and doubling or tripling the size of users can indeed be beneficial.

I will not even mention the network effects for the bipartite ad network (advertisers will choose a network with many content nodes, content nodes will put ads from a network with many advertisers). I found it rather ironic that Hal Varian, of all people, chose to ignore that aspect.

I understand the need from Google to play down the Microsoft-Yahoo agreement but sometimes it feels like listening to propaganda…

By: Cuando la escala de datos es irrelevante hasta en Internet | Denken Über

Cuando la escala de datos es irrelevante hasta en Internet | Denken Über — Fri, 14 Aug 2009 21:32:18 +0000

[...] Hay mucha data interesante en el blog de Nick Carr y datos en Search Engine Land y una nota excelente de Daniel Tunkelang [...]

By: Daniel Tunkelang

Daniel Tunkelang — Fri, 14 Aug 2009 20:03:25 +0000

Oops! Corrected now.

And I agree, what you measure and thus optimize for is critical. I still suspect that the bottleneck is creativity, not volume or scale.

By: jeremy

jeremy — Fri, 14 Aug 2009 19:39:45 +0000

But even there quality trumps quality

Do you mean “quality trumps quantity”?

how you choose what to test matters a lot more than how many tests you run.

And how you choose to measure what you test matters a lot more than either of those two factors. It’s a point that we discussed in your comments, before, with Max Wilson, etc. If someone runs a query and then doesn’t click anything, how can you tell the difference between that search being unsuccessful, because they didn’t find the information that they wanted, and that search being successful, because they found that the information that they wanted didn’t appear?

For example, someone queries for his or her name, and finds that the embarrassing drunk photo doesn’t show up in the first page of results. That’s a success! Or someone writing a patent does a bunch of searches for related work, and finds that there is nothing related. Success!

How does the search engine measure those scenarios? Does a 0.5%, 1% or even 2% experiment really give you enough data to tease out the difference in these two types of searches? Does a 100% experiment even give you enough data?