Category: General

I’ve been following the FASTForward ’09 conference from a distance–somehow I couldn’t justify spending a few thousand dollars to attend just to satisfy my curiosity, not to mention explaining a “business trip” to Vegas. Instead, I read what I could find online. Microsoft invited (I assume, paid) several bloggers to attend the event and write about it at the FASTForward Blog. Independently, a number of folks have been tweeting about the conference using the #ffc09 hashtag. And a few folks have been blogging about the conference independently.

A few observations:

• I’m not persuaded by corporate bloggers, and invited bloggers come across as corporate. Read the posts yourself at the FASTForward Blog, and I think you’ll agree that the fare, while not shrill advertising, is nonetheless a bit bland. Full disclosure: I participated in an effort last year to live-blog the Endeca Discover user conference. I even remember doing research on the fly to add context around the presentations. In retrospect, however, I think it was a misguided effort. No one wants to read a corporate blog.

• It’s cool that attendees get to hear from social media luminaries like Clay Shirky, Charlene Li and Peter Kim. I’d love to have listened to those presentations, though again there’s that price tag. Even remote attendees had to pay $495. What I wonder is why these speakers would be keynotes at a conference about enterprise search. One of the FASTForward bloggers, Sandy Kemsley, noted:
  

  Something that I noticed at last year’s FASTforward is that this is more than just a user conference: it’s also a social media conference. I don’t know how a search company’s user conference ends up like that, but it makes it interesting. What has changed since last year is that it seems that they’re required to inject the word “search” into everything in order to reinforce the overall message of the conference.

  I don’t doubt that “it makes it interesting”. But I do wonder if that’s what attendees signed up for.

• For all that, it sounds like there were some on-topic sessions, including a presentation by IDC analyst Sue Feldman (who is a great advocate for HCIR, even if she prefers the term “conversational systems“) and presentations by FAST executives, customers, and partners. And I imagine that, independent of the formal structure of the conference, much of the value of attending comes from the informal conversations with other attendees.

I’m curious to hear candid impressions from FASTForward ’09 attendees who find their way to this blog. I’d love to share what I learn with my co-workers who are organizing Endeca Discover ’09, as well as with colleagues who are organizing vendor-independent search conferences.

The great thing about public conversation is that everyone can learn from everyone else. Please do your part: educate me and keep me honest.

The news, which has been an open secret among enterprise search insiders for a while, is finally official: Microsoft announced at FASTForward ’09 that they will be bundling the FAST enterprise search product with their SharePoint product. To be more precise, it will be available as a $25K /server add-on for SharePoint customers. Considering the total cost of a typical SharePoint deployment, that’s practically free. Or, as Mark Bennett put it, an “aggressive price”.

Who are the winners and losers here? Unclear. Is this initiative aimed at Autonomy, Endeca, or Google? At all of the above? What are the implications for FAST’s legacy ESP 5.x customers, who have been promised 10 years of support?

My take: Microsoft sees SharePoint + FAST as a response to Autonomy + Interwoven, or vice versa–Microsoft’s acquisition predates Autonomy’s by a year. Both companies seem to believe that, in order to sell a search engine, they need to bundle it into a content management system.

Not everyone believes that. Google and Endeca are focusing on search as the main prize, not to mention a slew of smaller vendors. We may all disagree on the right way to crack the search nut, but we haven’t given up on building a better nutcracker.

Did Microsoft pay $1.3B just to upgrade SharePoint Search? I’m only a researcher, so perhaps I don’t appreciate the economics of the transaction. I’m just glad not to be the guy who will ever have to justify that decision to my manager, let alone a board of directors.

Today is the first day of FASTForward ’09, the annual user conference hosted by FAST (the enterprise search company acquired by Microsoft last year). I thought it would be a good day to reflect on the variety of search-related conferences and user groups that are competing for our attention and wallets these days.

First, there are the vendor user conferences. These include FASTForward, Endeca Discover, and a slew of events hosted by smaller vendors. These are great events for those vendors’ current customers, and occasionally are used as sales tools to persuade select prospective customers. At their best, they emphasize knowledge sharing among customers, as well as substantive presentations about the vendors’ products and services. At their worst, they offer a mixture of propaganda and entertaining (but not necessarily relevant) guest lectures. And they aren’t cheap: it’s $795 to attend Discover and $1,695 for FastForward–plus travel and lodging! Still, if you learn something that saves you a few days of consulting services, you’ll get your money’s worth.

Then there are the vendor-independent industry conferences. A few that are coming up:

• Enterprise Search Summit
• Infonortics Search Engine Meeting
• Text Analytics Summit

The good news about these is that, because they are vendor-independent conferences, you’re likely to hear a variety of perspectives. If they’ve selected their speakers well, you’ll learn about the different technologies even the philosophies underlying those technologies. If not, then be braced for a bunch of warmed-over sales pitches. Because that’s the bad news: it’s hard to run a for-profit search conference and actually make a profit–and no, these usually aren’t cheap either. The main sponsors are usually vendors and consultancies, which is both expected and appropriate. The sponsorship model only becomes a problem when the speakers earn their slots through sponsorship rather than through the merits of their content.

And then there are the academic conferences, particularly CIKM and SIGIR. The good news with these is that the content is top-notch: peer-reviewed presentations from top researchers around the world, hailing from both academia and industry–the latter usually representing the major industry research labs. They are also relatively inexpensive, since they are run by non-profit organizations and rely heavily on volunteers. The bad news is that research isn’t always immediately relevant to practice. Indeed, some of the presentations will make your head spin twice: first, as you focus to understand them, second, as you struggle to figure out how to apply their results or key insights to your real-world problems.

In my opinion, all of these leave a gap–a need for conferences that bring the rigor and seriousness of academia to bear on content that is relevant to industry practitioners. I’m hoping that the SIGIR ’09 Industry Track helps fill this gap. But I further hope that everyone in the conference-organizing business, regardless of their business model, shares the aspiration to deliver quality content that has real impact on the practical world of search.

This evening, I had the opportunity to hear Jeff Jarvis talk about his recently published book, “What Would Google Do?“. That opportunity was briefly in doubt: 277 people signed up for the event at the Daylife office, which had planned for a capacity of 150. Fortunately for me, my friend Ken Ellis let me in early, and I was not turned away at the door. Which is fortunate for you, since it means I have something to blog about!

Jarvis was entertaining, as expected. He is an excellent speaker, both when he’s delivering prepared material and when he’s put on the spot by aggressive audience members, who were in no short supply.

I perhaps deserve credit (responsibility?) for inciting the mob by asking the first question, suggesting that Google was the opposite of transparent (one of the most “Googly” qualities in his enumeration) and that, if we were to learn anything from Google, it was that success is best achieved through benign dictatorship. In fact, I told Jarvis that I thought he’d already seen the light on this issue.

Jarvis didn’t even flinch. First, he made clear that he was more interested in “the idea of Google” than the company itself. Second, he argued that Google has made the world transparent, even if Google isn’t always transparent itself. Finally, he suggested that being in continuous beta was a form of transparency. I didn’t have a chance to follow up after that, but others did, and I was happy to see that the crowd, on the whole, seemed unpersuaded by the culty premise of “the idea of Google”.

But what I enjoyed far more that the advertised event was the heated conversation I had, following Jarvis’s presentation. Bob Wyman, an engineer at Google, offered a full-throated defense of as many of Google’s technology and business decisions as I could question. While we ultimately agreed to disagree, I credit him for making a serious case. Unfortunately, I couldn’t take notes and uphold my side of the argument at the same time, so I’ll apologize in advance for any details I’ve lost or garbled in my good-faith recollection.

Here is what I recollect from our discussion:

• Our biggest point of contention was about the black-box nature of Google’s approach to relevance. Bob was quite familiar with the analogy to security through obscurity, but he objected that, before the discovery of public key cryptography, security through obscurity was the best game in town. In other words, the folks working on relevance ranking algorithms are still waiting for the equivalent of Diffie-Hellman or RSA.
• He rejected attention bond mechanisms as gameable, though I don’t recall any explanation as to why. It’s possible that he simply wasn’t familiar with them, and that my explanation didn’t do justice to the concept.
• He insisted that I wasn’t giving enough credit to Google for its experimentation–specifically, that I underestimated how much variation there was in result ranking  based on the collection of simultaneous experiments running at any given time.
• He felt I was being unreasonable to expect Google to disclose more about its retrieval approach, not only because it would help spammers, but also because it would unnecessarily give users more to think about.
• Finally, he felt that almost any clever idea would break down because of the combined constraints imposed by the hordes of spammers, the scale of the data, and the challenges of freshness.

Bob defended Google well, and I can’t say that either of us “won” the fight. Indeed, so much hinges on whether you can believe, as he does, that Google is only limited by what technology makes possible and what its engineers can implement. He rejects my assertion that Google’s has crippled the user experience because of some philosophical predilection towards black box approaches. In fact, he maintains that Google is incredibly open for a company of its size.

Unfortunately, the truth resides in the ultimate black box:  I can’t evaluate Google’s motivations from the outside. Bob invited me to work at Google to help solve the problem from the inside (I don’t think he meant that as a literal offer), but that not here or there. I think it’s only fair to judge a company from the outside. If Google wants to fix the misimpressions that I and others hold, it can certainly do so by providing more information. Absent such information from the source, I have to fall back on the public data and my powers of reasoning.

But let me say this clearly: I believe that most–perhaps all–Googlers mean well. Google’s China policy notwithstanding, I don’t think that Google is an evil company. On the whole, Google has done much more good than bad, and I believe that doing the right thing is a core company value, even if Google does not always live up to its aspirations.

My only problem is that, in the one area I’m most passionate about, it seems that Google is holding the world back. Google has become the legacy system that HCIR has to beat.

A parting joke, courtesy of Ken Ellis:

Q: What would Google do if they were a restaurant?

A: They would build a search engine and an internet ad auction system.

The other day, my wife told me a story that struck me as a great parable about exploratory search–that is, if true stories qualify as parables.

It was lunch time and she was facing a problem familiar to many of us city dwellers (well, New Yorkers at least): she wanted to find some place interesting to eat among the overwhelming set of options. She decided on a new approach–her version of asking “What Would Google Do?“.

She saw a couple exiting an office building and decided to follow them (at a discreet distance) to their lunch spot. She ended up at…McDonald’s. And no, I didn’t make this up, much as I might have tried!

While pop philosophers and psychologists have made much of the “wisdom of crowds“, there’s a dark side: crowds rarely come up with anything interesting. Outsourcing your decisions to a crowd may yield a satisficing decision, but don’t get your hopes up for more.

Granted, there’s more than one way to crowdsource: no one says that you have to go with a majority vote, or to follow someone at random. But leveraging the wisdom of a crowd in a more meaningful way requires a means to comprehend the diversity of views among the individuals that comprise that crowd.

In other words, you need exploratory search.

This Thursday, I’m attending an evening with blogger Jeff Jarvis that is part of his book tour for “What Would Google Do?” I took the opportunity to review an Intelligence Squared debate, hosted last November by the Rosencranz Foundation, about whether Google violates its “don’t be evil” motto. You can listen to the debate on the NPR web site or read the transcript here. But, for those who can’t spare the hour and a half to listen or who don’t care to read through a 74-page transcript, I’ll attempt to summarize here, and then offer my own thoughts on both sides of the question.

Six people participated in this debate, not including the moderator, ABC News correspondent John Donvan.

Arguing for the motion that Google violates its motto:

• Harry Lewis, Harvard, Professor of Computer Science
• Randal Picker, University of Chicago, Professor of commercial law and senior fellow at the Computation Institute
• Siva Vaidhyanathan, University of Virginia, Associate Professor of Media Studies and Law

Arguing against the motion that Google violates its motto:

• Esther Dyson, journalist and entrepreneur
• Jim Harper, director of Information Policy Studios at the Cato Institute
• Jeff Jarvis, blogger; associate professor and director of interactive journalism program at the City University of New York

Google was invited to participate, but chose not to do so.

First, since the game is over, let’s take a look at the score, based on polling the debate’s live audience.

Before the debate:

• 21% were for the motion that Google violates its motto.
• 31% were against the motion that Google violates its motto.
• 48% were undecided.

After the debate:

• 47% were for the motion that Google violates its motto.
• 47% were against the motion that Google violates its motto.
• 6% were undecided.

While this may seem like a tie, it was clearly a victory for those who changed the most minds–namely, those arguing that Google does violate its motto.

But we at The Noisy Channel are not sheep who uncritically accept the purported wisdom of crowds. Let’s look at the actual arguments. I will try to distill the main arguments made by each side.

Those arguing in favor of the motion, i.e., that Google does violate its “don’t be evil” motto, made what amount to four  points:

• Google’s China policy is evil (Harry Lewis): Google’s collaboration with the Chinese censors makes them complicit in the brainwashing and thought control of the Chinese people. Google’s motives cannot be equated to those of, say, the Voice of America in Bulgaria, even if both resulted in oppressed people experiencing partial access to the truth. The VOA was subverting Bulgaria’s regime, while Google made the deliberate choice to do business with a country whose local laws would require them to do evil. Google chose to collaborate with China’s censorship regime. That deliberate act violates its motto.
• Google is evil by its own standard (Harry Lewis): Google surely intended a higher standard by “don’t be evil” than “don’t be as evil as Hitler”. Moreover, aspiration to not be evil isn’t enough. Google doesn’t live up to its aspirational promise. Google should be held to their literal motto, not the standard of whether they are evil in their hearts.
• Google does evil by monopolistic abuse of market power (Randal Picker): Google has designed its auctions in a way that takes advantage of its market power. When faced with a conflict between what’s good for the world and its own interest, Google favors its own interests. Google acts as a monopoly, and its monopoly power is at the heart of its business model. They create evil that they do not need to create.
• Google is guilty of hubris (Siva Vaidhyanathan): Google sees no limits to its power, and thus commits the deadliest sin of hubris. Aggressive competitiveness doesn’t make Google evil, but its holier-than-thou attitude is hypocritical hubris. Comparing Google to peers is irrelevant: Google has set its own standard and has not met it. Google’s motto reflects its hubris and is a cynical marketing ploy, a promise it has not kept.

Those arguing against the motion made these five points:

• Google’s China policy virally spreads democracy (Esther Dyson): Google spreads democracy virally by its very presence, by engaging rather than succumbing to apathy. Google is not collaborating with the Chinese censors so much as infiltrating the regime, exposing people to the virtues of knowledge, inciting change for good. Google is bringing goodness to the world, eroding power structures and the abuse of power by others.
• Google is not like other greedy corporations (Esther Dyson): Google’s leaders do not hold themselves directly accountable to shareholders, but rather pursue their own agenda that they believe maximizes long term value to shareholders. Google has set its own standard and has met it by aspiring to do good.
• Much of the evil attributed to Google is more correctly attributed to governments (Esther Dyson): The real danger is not Google, it’s the government. Governments have power which can easily be abused, while Google is constrained by law, by competition, and by its users. Those constraints actually help prevent Google from being evil, but beyond that Google doesn’t want to be evil. In any case, don’t blame Google for bad things it does in order to comply with the law.
• Google is no more evil than the internet as a whole (Jim Harper):  Google is neither evil like Hitler, nor does the fact of its aggressive competitiveness make it even in the Enron sense of being driven by corporate greed. Google is essentially good, and its stated aspiration to not be evil argues in its favor.
• If Google is evil, then humanity is evil (Jeff Jarvis): Google is good and virtuous, and therefore not evil. Its very aspiration to not be evil is evidence in its favor. If Google is evil, then the standard is too high and we are all evil.  Why Google acts is more important than how Google acts, and Google acts out of the belief that they are trying to do good. If that is not enough the standard is God-like perfection. Google’s only crime is its success.

These are interesting arguments. But let me try to offer two arguments of my own to each side that, in my opinion, strengthen their positions:

For those arguing that Google is evil:

• Google’s collaboration with the censors in China is comparable–albeit to a much lesser degree–to the behavior of companies that conducted business with the Nazis or supplied technology to the apartheid regime in South Africa. Google provided technology that the Chinese government wanted–perhaps even needed–in order to ensure China’s global competitiveness. But even if China did not need Google’s help (after all, they have Baidu), Google still was evil to collaborate with a regime that oppresses its people by censorship and widely documented abuses of human rights. Imagine walking in on a mob of people beating someone to death. Your contribution might have no effect on the outcome of the hapless victim. But it is still evil to participate.
• Google is a parasite, abusing its market power to dominate the retail, media, and advertising industries. Two specific examples: the “secondary search” feature that Google imposed on retail and media sites, undermining those site owners’ control of their users’ experience; and the lack of transparency in Google’s auction model for advertising. Google’s successful acquisition of DoubleClick and its abortive attempt to partner with Yahoo! show that it only seeks to reinforce this market dominance.

For those arguing that Google is not evil:

• If Google’s collaboration with the censors in China is evil, it is at most a lesser evil. China already has Baidu, which offers comparable technology and competitive advantage to China as a nation. Google at least offers a wedge that might open up China to viral democratization. As Google senior policy counsel Andrew McLaughlin said, Google made the decision that provides the greatest access to information to the greatest number of people.
• Google may be not be winning the love of the retail, media, and advertising industries, but it is serving the interests of its the constituency that matters: users. Google has given users–ordinary people, the little guy–unprecedented access to information, and has unlocked the control that retailers and media companies have held over this content. Furthermore, Google has democratized advertising, making it possible for smaller players to operate in a market previously dominated by big, mass-market brands.

And where do I stand personally? I think it’s silly to evaluate a company in terms of good and evil, even if Google has invited us to do so. There are the rare examples of companies being evil (IG Farben in the traditional sense of the word; Enron in the more modern sense of being the caricature of a morally bankrupt corporation), but these are the exceptions. I find it more useful to ask whether Google is making the world a better place.

For the most part, I think the answer is yes. I remember the pre-Google world, and I’m much happier living in this one. Nonetheless, I believe that Google, by resting on its laurels, is dragging the rest of the world down it its complacency. Do I blame Google for its complacency? Yes, I do, and I’ve told them as much.

But I also blame the rest of us. We continue using Google by choice–perhaps pressured by network externalities, but certainly not by coercion. It’s up to us, as users and as technologists, to set the bar higher.

Perhaps Amazon has had this feature for a while, but today, for the first time I noticed a section labeled “Customers Who Bought Related Items Also Bought” as seen in the screen shot above. I was looking at an unreleased book, which might explain why they couldn’t show me information based on customers who actually bought the item.

Has anyone else noticed this? Am I just late to the party? I tried to find more information online, but nothing showed up. I assume they are using some item similarity measure to assemble a set of related items, and then are basing collaborative filtering on the purchase history associated with that set.

I’m very curious to hear more from anyone who is familiar with this functionality.

I’m a long-time fan of ad blocking software, from the Siemens Webwasher plug-in in the early days to the Adblock Plus and CustomizeGoogle Firefox add-ons today. I know that some people view the use of ad-blocking software on ad-supported sites as anti-social or unethical. My personal view is that it is no different from physically obscuring the ads, or muting a television set during an advertising break. In any case, the technology allows it, and I’m am a very satisfied customer.

But I’m delighted to see that mainstream sites are finally starting to understand that advertising should not be coercive. Check out a post by Marisa Taylor in the Wall Street Journal entitled “Turning the Ads Off“. There she notes that some high-traffic sites, including wikiHow, AboutUs and, Kayak.com, are allowing users to opt out of ads.

Most users apparently aren’t opting out. According to Jack Herrick, founder of wikiHow:

“‘Opt-out’ ads are the good netizen thing to do for users,” he said. “It doesn’t actually hurt revenues that much anyway. And users love it. So why not do it?”

The other day, I asked a friend at Google if, given the option, she’d opt out of Google’s ads. Since she’s an employee, I imagine she might take the site’s terms of service seriously, or at least have more moral qualms about violating them. But she said that she found value in the ads, and wouldn’t opt out of them. Indeed, Google claims that the ads are valuable to users, not just advertisers.

I’d love to see Google put its money where its mission is, and make it easy for users to opt out of ads. That would show true leadership, as well as a confidence in its most sacred principle: “Focus on the user and all else will follow.” But I’m not holding my breath.

According to Wikipedia:

Pollution is the introduction of contaminants into an environment that causes instability, disorder, harm or discomfort to the physical systems or living organisms.

Wikipedia does not supply a definition for “cyber pollution”, but Ragy Thomas offers one in a recent post: “the waste of time and energy created by the inconsiderate use of commercial and personal electronic communication.” Thomas is no stranger to the subject: he has held advisory and leadership positions at Epsilon Interactive and Goodmail Systems, which makes him familiar with both the creation and prevention of cyber pollution.

I’ve advocated for attention bond mechanisms (ABM) in the past, and I still feel they are the most promising weapon against cyber pollution. In fact, Goodmail is a step in the direction towards implementing an mainstream ABM. But Thomas cites a service that is truly an ABM: Rupeemail. As per the Rupeemail FAQ:

Merchants and advertisers send RupeeMail to recipients like you seeking your attention. RupeeMail can only be sent to recipients who have explicitly agreed to receive RupeeMail by either registering at the RupeeMail website or permitting a specific sender to send email to them.

Merchants and advertisers send RupeeMail either to a list of willing recipients selected from the RupeeMail data base or to their own list of customers that have opted to receive email from them. If the recipient opens the the RupeeMail, then the recipient collects the value of the stamp attached to the RupeeMail. The sender (advertiser) decides the value of the stamp paid to recipient as a gesture of appreciation for recipient to open the RupeeMail and hopefully read their message (advertisement, news letter, survey etc.)

I’m thrilled to see people employing approaches like these in the war against attention terrorism. Spam filtering, despite its successes, is neither sufficient nor necessary. What we really need are communcation mechanisms that reflect true attention scarcity and ensure the negotiaton of attention costs between the sender and the recipeint is mutually satisfying. Otherwise, as with all pollution, we have a tragedy of the commons that buries us all in crap.

What happens to a stream deferred? Does it dry up like a raisin in the sun? No, it eventually makes it on to YouTube! Despite some early difficulties, Google has managed to make my recent Tech Talk, “Reconsidering Relevance”, available on YouTube. My apologies to all for the delay. Slides available on SlideShare.