SIGIR 2009: Day 3, Industry Track: Analyst Panel

The morning sessions of the SIGIR 2009 Industry Track consisted of five individual presentations; the afternoon consisted of two panels. The requirement to synchronize with the research talks led to the allocation of 90 minutes for each panel–which was a bit more than I’d originally planned on (and this change, like many, occurred in the two weeks before the conference). James Allan, one of the SIGIR 2009 co-chairs, suggested that we add an academic responder to each of the panels to account for the additional time, and we went with that approach.

The first of the two afternoon panels consisted of industry analysts: Whit Andrews (Gartner), Sue Feldman (IDC), and Theresa Regli (CMS Watch). I moderated the panel–or, more accurately, attempted to moderate it. Marti Hearst served as the academic responder.

The panel opened with each of the three panelists making an opening statement, sharing their perspectives about the key business concerns and trends in the search industry. I asked them to talk about enterprise search in the broadest sense of the term–search applications that companies buy or build)–rather than in the narrow sense of no-frills intranet search.

It became immediately clear from their opening statements that the panelists had wildly different perspectives and styles. While I think the term “food fight” that I heard bandied around afterward is a bit of an exaggeration, they certainly engaged in a heated debate.

One topic that attracted particular controversy was how enterprise search applications should assign relevance to search results. Whit suggested that, in an enterprise setting, the main objective function is the profitability of the enterprise, and that relevance should essentially be money driven. Sue and Theresa disagreed sharply, mainly arguing that relevance should be user-controlled.

I’m probably oversimplifying their arguments, and in any case shouldn’t take sides in a debate among analysts! Still, Whit probably won’t be surprised that my sympathies generally lie with the users. That said, Whit is right that enterprise search companies sell to enterprises, not directly to customers, and those enterprises (like web search companies that sell to advertisers) may have interests that aren’t always aligned with those of users. At Endeca, we advise our customers on how to configure and communicate a relevance ranking strategy, but ultimately our customers make their own decisions. After all, it’s their site and their money.

And that leads to the other topic that caught my attention and came up during Marti’s responder session: the question of how analyst firms make money. All three of the panelists were open about how their employers make money, whether from enterprise buyers, vendors, or some combination thereof. My personal preference would be that analysts make money primarily from enterprise buyers–but of course I work for a frugal vendor. I’ve heard from a variety of sources that Endeca “doesn’t spend enough” on analyst services–or on corporate marketing in general. Since neither vendors nor analyst firms open up their books, I can only speculate. Fortunately, it’s clear the analysts on the panel not only have integrity, but also have strong enough views that they can’t probably couldn’t be swayed by money or pressure.

As organizer of the track, I indended for the panel to offer an audience of mostly academic types a chance to see people whose opinions influence tens (if not hundreds) of millions of dollars in purchasing decisions. I hope I accomplished that. I’m especially grateful to these highly billable analysts for freely sharing their time and ideas. Neither SIGIR nor I could possibly have afforded their market rates!

For other perspectives, take a look at Theresa’s blog post, “Know Your Relevance“, or Mary McKenna’s summary post.