Agreed. Caveat, however: transparency alone wouldn’t be enough. Because not all queries (e.g. navigation) need transparency. So the independent search engine that we’d need to build to run this experiment has to be equal in quality to Google for navigational queries, with the ability to switch to transparent interaction for informational/exploratory queries. Given that requirement, Wikia Search does not count as such an experiment.

“But the success of Hadoop that I alluded to earlier is evidence that this transparency isn’t completely obscured.”

Yup, I agreed to the Hadoop/Mapreduce benefits in my comment #8, above. Still, that’s not a part of the system (other than query response time) that the user sees. I’m interested in HCIR.

“I’m glad we agree that we’re debating shades of gray.”

Of course. We’re not republicans, now, are we?

“I agree with colleagues that there’s a real risk to users in over-disclosing the details of ranking. I hope to see the day where it’s possible to have relevance without obscurity.”

One day when you can finally, legally talk about it, I’d like to know what it is you’ve learned since joining the Big G that seems to have moved you more toward this risk-averse state. Maybe that’ll have to be in 20 years. But I have sheer academic curiosity about the thinking process that is occurring.

As for the research papers, I think Google has made a point of promoting what it believes to be the most important ones–and they are infrastructure papers, not search papers per se. But the success of Hadoop that I alluded to earlier is evidence that this transparency isn’t completely obscured.

I’m glad we agree that we’re debating shades of gray. I’d like to strive toward a lighter shade than the current one, but I agree with colleagues that there’s a real risk to users in over-disclosing the details of ranking. I hope to see the day where it’s possible to have relevance without obscurity.

Matt’s general argument isn’t that there are things about Google that are open/known. His argument is that Google the company actively has an agenda to make more things transparent, carries through with that agenda, and that Google should get more credit for making things transparent (the active followthrough) than its critics give it.

Because PageRank was disclosed by NSF-funded grad students doing their job of publishing, and not by Google, it doesn’t support Matt’s argument at all.

I’m sorry that I’m being such a stickler about this point, but I see it quite strongly. What we’re talking about is what Google as a company is actively doing.

For that matter, Cutts writes: “Google has continued to publish literally hundreds of research papers over the years. Those papers reveal many of the “secret formulas” for how Google works”

I’ll ask the question again: “And sure, there are hundreds of papers by Googlers. But how many of those papers were done (primarily) by summer interns, who already came to Google with a solid sense of a research idea? And of the ones that are Google-internal only, how many of those are on production-code systems, vs raw research? Microsoft publishes hundreds of papers as well, the vast majority of which are never incorporated into shipped products. Does that also make Microsoft transparent?”

In other words, how much of this is “transparency through obscurity”. I.e. if there are hundreds of papers with secret formulas in them, and only 7 of those formulas actually make it into production code, and it is never make known which 7 those are, then how can this reasonably be called transparency? If there is so much flak thrown up, then any real disclosure is essentially so obfuscated so as to not really be transparency at all.

I do agree with you, Daniel, that it’s not a black and white issue. Google doesn’t get an A+ on transparency, nor do they get an F-. But the picture is also not quite as clean as Cutts paints it, either.

How much of this is chicken/egg? I remember Steve Jobs claiming that there was no evidence that users wanted to watch video on an extremely small screen. Six months later Apple released the video iPod, and now lots of people do.

If users don’t know or don’t understand what is possible, they’ll never ask for it. There has to be some education, first. How much education is Google doing?

But you raise an interesting point. I’m with you on the HCIR perspective of focusing on end-user penetrability. And Gregor is a refreshing example of a user who cares about the details of how the search engine works.

But I see far greater concern about the ranking details by site owners than by users, e.g. a lot more literature about site owners can improve their site ranking than on how users can search more effectively.

According to the Christensen’s theory of disruptive technology (at least Wikipedia’s explanation), ” a disruptive technology may enter the market and provide a product which has lower performance than the incumbent but which exceeds the requirements of certain segments, thereby gaining a foothold in the market.” If users value transparency, then there should be room for someone to offer a product that lags the incumbent on most measures but offers greater transparency. I believe this is already happening in markets like enterprise search, but I don’t see the evidence for it in web search.

And, to be clear, I’ve advocated for it! But that’s doesn’t make it so.

With all respect, I still strongly disagree about giving even a little credit — not to L&S — but to Google. You have to look at the motivation of Larry and Sergey, and what was going on at the time. At the time that they actually submitted the paper for publication, they were researchers at a university and funded by NSF money. Their job, their primary product, their deliverable, was publications. So when they published PageRank they did so before they ever started Google the company, and did so because it was the mandate of their academic position and government funding to do so. Not because of any attempt to be more or less transparent. Had they not published the paper, they would have not been properly doing their jobs at that time. Transparency had nothing to do with it.

I hear what you’re saying; they are indeed the founders of Google. But I still do not see how you can give Google credit for something that was a requirement of a pre-Google job. Doesn’t make any sense.

But I think Gregor gets to the heart of this issue, above. What Gregor is saying is that it doesn’t really matter what the exact mathematics of the Google ranking algorithm is. What matters is the end-user penetrability of that algorithm. Is Google transparent enough so that a searcher using the Google system can give Google the right signals, in the right way, at the right time, so as to find the information that is needed? And I think the answer to that is still no. It’s changing, and I do see small HCIR signs here and there. But at the end of the day, that’s the question that really matters.

Of course, making Google more HCIR-transparent will most likely involve exposing, in some fashion or other, more of the Google ranking algorithm. It might not be a raw exposure, but there will still have to be an exposure of some sort in order to give the users more control over their experience.

So the question: How transparent is Google in that regard?

From that perspective, all those papers on MapReduce (which I do give transparency credit for) and PageRank (which I don’t give transparency credit for — it was their job at the time!) don’t matter one way or the other. Because the technical details contained in those papers are still not exposed to the end user in a transparent way. There is no way for the end user to say “Stop using the PageRank popularity signal for my query, because I know that what I am looking for is in the tail!”

Transparency is therefore still at a minimum for what really matters: End user experience.

In any case, I agree that it would be great for researchers to run different algorithms and approaches on a Google-size index and with real-life queries. I think researchers are in a position to build a sufficiently large index–in part because of the publications I describe above. But what researchers really want–at least from what I have heard–is access to query logs and actual user traffic. That raises major privacy concerns for users, as well as concern about abuse by spammers. Don’t forget what happened a few years ago.

Anyway, there are good reasons for obscurity for the commercial and search quality reasons you mentioned. However, in order to advance scientific understanding of search quality, there is also a need for transparency. I mean that it should be possible for IR and HCIR researchers to run different algorithms and approaches on a Gooogle-size index and with real-life queries, and publish the algorithms and results. It would be nice to see Google contribute to such an endeavour, in order to prevent a situation where the knowledge about how to deliver a great search experience is confined to only a handful of big companies.

And yes, the measure doesn’t conform precisely to the paper–and in any case it’s only one factor. But, to put it in the context of Matt’s post: “One of the most widely-discussed parts of Google’s scoring has always been PageRank. That ‘secret ingredient’ is hardly a secret.”

As for the papers, MapReduce, the Google File System, Bigtable, and Protocol Buffers are essential production tools. I agree with Matt that Google deserves credit for enabling Hadoop, thus supporting its largest web search competitor.

In any case, no one at Google denies keeping the precise details of ranking secret. But I’d say that critics who extrapolate that the entire ranking algorithm is a black box overstate their case, given how much has been disclosed. Unless you feel that being transparent is like being pregnant–an all or nothing deal. I don’t believe that.

Of course, the other issue is Google’s motive for what secrecy it does keep. As per an official blog post by Google’s counsel, Google stands accused of demoting the positions of competitive sites. I don’t personally believe this to be the case, nor have I even seen anyone present evidence for it. I do understand how some people–particularly site owners unhappy with their ranking–may feel Google is guilty unless it discloses everything in order to prove itself innocent. Given the stakes, I don’t think it’s reasonable to expect Google to make such a sacrifice–at the expense not only of its own competitive position but also of its users.

Jeremy, I know you have many points of disagreement with Google’s approach, particularly with regard to transparency. Perhaps the extent of Google’s disclosure isn’t enough to earn a passing score in your book. I’m hardly giving it an A+. But I hope we can agree that it isn’t black and white.

So it is disingenuous, at the very least, to give Google credit for this. This information was released (at least submitted for publication with the intention of being released) before Google the company existed. So whatever L&S’s attitudes about transparency were as grad students, I think it changed once they became incorporated as Google.

And I cannot count the number of times I’ve seen Google talks in which the Google speaker has intentionally gone out of his or her way to mention that the form of PageRank that they use in today’s engine is nothing like the one published in the paper. I don’t know if that’s the truth, or a deliberate attempt at obfuscation. But the upshot is that this means that Google is even less transparent than claimed, because it has the feel of trying to throw people off track — now one doesn’t know what to believe anymore.. the published paper or the research talks.

So to go back now and try and claim “transparency credit” for this paper… makes me uneasy.

And sure, there are hundreds of papers by Googlers. But how many of those papers were done (primarily) by summer interns, who already came to Google with a solid sense of a research idea? And of the ones that are Google-internal only, how many of those are on production-code systems, vs raw research? Microsoft publishes hundreds of papers as well, the vast majority of which are never incorporated into shipped products. Does that also make Microsoft transparent?