Category: General

Sometimes the response to a comment is worthy of an entire post, and this is one of those times. In response to my recent post about Able Grape, a wine search engine developed by Doug Cook (now Director of Twitter Search), Lee asked:

Let’s say I know almost nothing about wines/digital cameras/cars and a search site offers me “options” to drill down. However, I can’t use those effectively and eventually it comes down to availability and price for me. My questions are what are your thoughts on these kinds of situations and is there a scientific explanation/theory on this case?

This may be why Google does not endorse faceted search except for experimental projects.

It’s a great question. There’s been a lot of research on how people make decisions when they have to manage trade-offs among multiple attributes, and the increasing interest in behavioral economics since Daniel Kahneman won the Nobel Prize in 2002 has helped some of that research has even percolated into the mainstream thanks to bestsellers like Freakonomics and Dan Ariely’s Predictable Irrationality.

The short answer is that there’s no point in offering users options that they can’t (or won’t) use effectively. Choice overload is certainly a problem, and our reaction to it is to satisfice, typically resorting to “fast and frugal” heuristics that throw out most of the potential decision criteria and instead focus on one or two attributes, e.g., price and availability.

But that’s no reason to dumb down the data we make available to decision makers. We make hard choices all the time, and fast and frugal can be horrendously suboptimal. We don’t hire employees based solely on their price and availability–or at least good employers don’t! For that matter, I don’t think most people pick wines that way, given that even Trader Joe has to diversify beyond “Two Buck Chuck“. And, while there’s probably more of a market for cheap cameras and cars, I’m pretty sure you’re an extreme outlier if you completely ignore other criteria.

That said, there are some caveats about exposing options to users. Faceted search is hard, especially on the open web. Take it from the folks at Microsoft Research–but I’m sure Googlers would be the first to agree, especially given their experience with projects like Google Squared that, while promising, are nowhere near ready for prime time.

I appreciate that Google is conservative about embracing faceted search–and HCIR in general. I’m actually impressed by the steadily improving quality of their related terms for search queries–even if they do hide them behind two clicks (show options -> related searches). Perhaps they’re feeling some pressure from Bing. But I think they’re largely following the dictum of “if it ain’t broke, don’t fix it”. Google is an extremely successful company. And, as Clayton Christensen argues, successful companies are great at incremental innovation and bad at disruptive innovation. As far as I can tell, faceted search is very disruptive to their model.

While I am an avid Twitter user (and apparently a tradeable commodity in a “Fantasy Twitter” game that some friends are playing), regular readers know that I’ve offered mixed reviews of Twitter Search.

I’ve link-baited Summize founder and Twitter Chief Scientist Abdur Chowdhury here once or twice, but I understand that he’s no longer running Twitter Search. They’ve got a new guy, Doug Cook, as Director of Search.

This is great news, because Doug is someone who’s thought a lot about search and user experience. He was one of the early web search guys at Inktomi and also spent some time at Yahoo!, but what impresses me most is a project he’s pursued as a labor of love: Able Grape.

From their about page:

We’re a wine search engine — not for comparison shopping, but for learning and research. We aim to be the world’s most comprehensive, up-to-date, and authoritative source for online wine information.

Great, another vertical search engine, just what the world needs (unfortunately WordPress 2.8.4 doesn’t support sarcastic font). But seriously, Able Grape is worth a look, even if, like me, you are not a wine nerd. So wash your glasses and let’s have a quick tasting.

First off, Able Grape is not searching a proprietary document collection. Rather, it’s based on a focused crawl of “more than 38,000 sites and some 18 million pages.” In other words, Able Grape is in no position to ask anyone to add meta-data. Even at the site level, I doubt Doug had the time to customize the handling of content for each of 38,000 sites. In other words, there’s enough scale here to make the problem interesting.

Now let’s look at some examples of the site in action. I’m a fan of Spanish wines, so I’ll start with one of their example queries, tempranillo. The first page of results looks relevant to the topic, but so far that doesn’t distinguish them from Google, Yahoo, or Bing. What surprises me is that the “Filter by Region” offers regions outside of Spain–like California and even New York! Yes, I might have learned some of that from Wikipedia–though it would not even have occurred to me to ask about non-Spanish Tempranillo. That’s exploratory search and serendipitous discovery for you!

Let’s try a different example, this time not from their list. I like Malbec wines (which I associated with my maternal link to Argentina), but the only local wine region for me is the North Fork of Long Island. So here’s a search for north fork malbec, filtered to Long Island. It certainly gives me ideas of which wineries to check out on my next trip there. Though, to be fair to the competition, G/Y/B all handle this query pretty well–though none of them offer refinement by region to disambiguate “north fork”.

Able Grape has lots of cool features, ranging from how they handle multilingual content to clever use of constrained “wildcard” terms like anyvariety to match any wine variety (aka varietal). I suspect that there is much to learn from its design that applies to a broad variety (sorry!) of search applications.

I’m a wine dilletante, so it’s hard for me to spend too much time on this site without any deep-seated information needs to fulfill. But I’m a card-carrying member of searchaholics anonymous (well, maybe not so anonymous), and I’m impressed by what Doug’s done with this vertical.

Which brings us back to Twitter Search. Director of Search for Twitter is a high-profile, high-pressure job, even without Facebook nipping at Twitter’s heels. I’m sure Able Grape will ferment for a while as Doug devotes his creative energies to improving Twitter Search. I certainly hope he brings the same focus and sensitivity to his new endeavor and makes Twitter Search a grand cru of search engines.

I try not to write posts that are just cut-and-paste from Techmeme, but it’s hard to resist a trio like this:

• Facebook rolls out new version of search
• Facebook acquires FriendFeed
• Google testing new “caffeine” web search infrastructure

OK, perhaps that last item isn’t strictly search news, but it may as well be, given that the microblogging wars are in no small part about “real time” search.

I’m not a huge Facebook fan (as those of you who have looked at my spartan profile page may have noticed), but I am curious about how they’re implementing search over their sprawling collection of content. I’m underwhelmed with my own search experience on the site, but that might be my own fault for not being an active Facebook participant. Perhaps folks here who are more active can share their own experiences.

As for the acquisition of FriendFeed, I’m surely in good company to assume this was Facebook’s second choice after the attempt to acquire Twitter fell through. If, as has been reported. Facebook only paid $50M for FriendFeed, then the acquisition was pocket change compared to the $500M they offered Twitter  (granted, some or all of that being based on a controversial valuation of the Facebook). Anyway, it should keep life interesting in the status-sphere.

And then there’s Google’s preview site, which you can try here. The only difference I see between it and the non-preview Google search is that the estimated result counts tend to be slightly higher. The top-ranked results seem almost identical, modulo tiny permutations for the queries I checked, as do related searches and any other features I tried. But apparently that’s the idea:

The new infrastructure sits “under the hood” of Google’s search engine, which means that most users won’t notice a difference in search results. But web developers and power searchers might notice a few differences, so we’re opening up a web developer preview to collect feedback.

Anyway, it’s more fun reading all of this stuff than hearing the CEO of the web’s great search brands proclaim that her company has never been a search company–or wondering where all the great search people I know there will land as Yahoo search is assimilated into Bing. Don’t get me wrong, I’m looking forward to the competition between Google and Microsoft–one that I think will finally be waged in earnest. But I’m still sad for Yahoo and its employees.

Which bring us to the last news item: Doug Cutting is leaving Yahoo for Cloudera, where he’ll continue to work on Hadoop. According to his blog post about it, “This move will not fundamentally change my day-to-day activities.” It will certainly be interesting to see what comes next from someone who has been instrumental to so many major open-source packages associated with search.

The other day, I was talking with Paul Thompson about the challenges of evaluating interactive information retrieval (IIR) systems, and he mentioned a paper that came up in discussion at the SIGIR 2009 workshop on Understanding the User: “A probability ranking principle for interactive information retrieval” by Norbert Fuhr–an update to the decades-old probability ranking principle (PRP) for information retrieval.

I was embarrassed to admit that I’d never seen or even heard of this paper, despite Nick Belkin citing it in the ECIR 2008 keynote that inspired my first blog post! In my defense, the citation is a single sentence that offers more of a tease than an explanation of the paper’s thesis.

Have no fear; I’ll offer more than a sentence here! I’ll summarize the paper and then offer my personal reaction. Let’s start with a few lines from the abstract:

In this paper, a new theoretical framework for interactive retrieval is proposed: The basic idea is that during IIR, a user moves between situations. In each situation, the system presents to the user a list of choices, about which s/he has to decide, and the first positive decision moves the user to a new situation. Each choice is associated with a number of cost and probability parameters. Based on these parameters, an optimum ordering of the choices can the derived – the PRP for IIR.

The first two sections of the paper provide an introduction and motivation. I’ll skip these, other than excerpt the following:

Interactive retrieval consists of user actions of various types, and scanning through document lists for identifying the relevant entries is not the most crucial activity in interactive retrieval [Turpin & Hersh 01]. In contrast, other activities (like e.g. query reformulation) seem to be more ‘expensive’ from the user’s point of view.

It’s in the third section that Fuhr starts outlining his approach. He established three requirements that, in his view, a PRP for IIR must satisfy:

• Consider the complete interaction process.
• Allow for different costs and benefits of different activities.
• Allow for changes of the information need.

He then makes four simplifying assumptions:

• Focus on the functional level of interaction (i.e., ignore design, visualization).
• Decisions are the major interaction activity.
• Users evaluate choices in linear order.
• Only positive, correct decisions are of benefit for a user.

With these assumptions in hand, Fuhr establishes an interaction model comprised of a sequence of “situations”:

a situation reflects the system state of the interactive search a user is performing…a situation consists of a list of choices the user has to evaluate…the first positive decision by the user will move him to another situation (depending on the choice he selected positively)…assume that there is always a last choice that will move him to another situation…the user’s information need does not change during a situation, knowledge is added only when switching to another situation due to a positive decision

In the fourth section, Fuhr describes a cost model for IIR. The key concept is that each choice incurs an effort, yields an expected benefit, and incurs an additional cost if the user has to backtrack. His overall framework is generic, but he offers a concrete “illustrating example” in which:

• queries represent conjunctions of query terms
• choices represent narrowing query refinements that add individual terms to the current query
• the probability of the user choosing a given query term is proportional to that term’s frequency in the corpus
• the benefit of an accepted choice is the log of factor by which it reduces the result set

Given such a cost model, the goal of an IIR system is now to maximize the user’s overall expected benefit, which in turn requires making tradeoffs at each choice. Fuhr offers an example to illustrate these tradeoffs:

As a simple example, assume that the system proposes some terms for query expansion. As one possibility, only the terms themselves are listed. Alternatively, for each term, the system could show a few example term occurrences in their context, thus giving the user some information about the usage of the term. The user effort per choice is lower in the first case, but the decisions will also be more error-prone.

The fifth section explains optimum ranking of choices according to the PRP. It has the most math, and I’ll refer the interested reader to the paper. The section that intrigued me the most was the sixth, entitled “Towards application”, in which Fuhr considers the various components of his generic framework, and speculates how existing and future research may supply the concrete details needed to instantiate it.  Finally, Fuhr wraps up  by citing related work and offering a conclusion/ outlook.

That’s the summary–perhaps it will inspire some of you to read the whole paper. Personally, I’m intrigued by Fuhr’s model.  I suspect I was trying to back into a similar approach in my post on “Ranked Set Retrieval” a few months ago. But, even if I’d try to formalize my proposal, I don’t believe I would have ever arrived at a framework as general as Fuhr’s.

That said, Fuhr’s article is like a fancy dinner that leaves me hungry at the end. It’s a solid paper: general framework, suggestive examples, and good writing generally. Nonetheless, it doesn’t give me what I really wanted: to feel closer to either a cost-effective evaluation methodology for IIR–or a tool that can be embedded into an IIR system to improve user experience. I know it’s a theory paper, and I should judge it on its own terms, but I was hoping to see more immediate applicability.

Given that Fuhr has published a fair amount of applied research, I’m hopeful about his future work in this area. I do appreciate the formal framework he has proposed, and I’d like to think that people are already looking at way to build on that framework. Perhaps I’ll be one of them.

A colleague just sent me a link to a story about a Twitter user being sued for a tweet. At least he’s not being sued in London.

I’m strongly if not absolutely in favor of freedom of expression, so it’s hard not to find such cases depressing. Nonetheless, I don’t think  the legal landscape hasn’t changed.

Rather, what has changed (or accelerated) is that:

• It is easier for people to express themselves publicly–and hence far more people are doing it.
• The detached nature of online communication releases people’s inhibitions. Moreover, people not only don’t self-censor, but in some cases are deliberately provocative to attract attention.
• The speed and efficiency of distribution (especially through search / alerts) means that the people most likely to be or feel damaged by an act of public expression are far more likely to discover that act.

So it’s not surprising that users are being sued for what they say online–it’s an expected consequence of the democratization of publishing, especially in the litigious English-speaking countries on both sides of the pond.

I’d personally like to see it a higher bar for someone to initiate a defamation lawsuit–let alone win it–but I’m not holding my breath. Instead, I expect that we’ll see more anonymous expression by people who don’t feel the authenticity of disclosure justified the risk of retaliation. Oh well.

The last session of the SIGIR 2009 Industry Track was the enterprise search vendor panel. Originally, I’d hoped to have CTOs (or the equivalent) from Autonomy, Endeca, and FAST–specifically, Peter Menell (CTO of Autonomy), Adam Ferrari (CTO of Endeca), and Bjørn Olstad (formerly CTO of FAST, now a Microsoft Distinguished Engineer).

Since it would have been inappropriate for me to moderate a panel that included my own manager and representatives of two of Endeca’s competitors, I recruited Liz Liddy, who not only is the chair of SIGIR, but also whom I felt was uniquely qualified to understand both the research and business sides of this field. As if that wasn’t enough, James Allan managed to procure Bruce Croft, whose volume of achievements includes both a Salton Award and a Research in Information Science Award, the highest honors in information retrieval and information science. And I recruited our very own commenter-in-chief Jeremy Pickens to serve as a time-keeper. I wish I’d also enlisted him for the panel I moderated!

That, at least was the plan. Plans, of course, are subject to change. Only a couple of days after I reached out to Bjørn, I saw that he was promoted to replace former FAST CEO John Markus Lervik. He suggested Øystein Torbjørnsen, chief architect of FAST’s core search. Fine by me. Two weeks before the conference, however, Bjørn wrote to inform me Øystein had to back out for personal reasons. But he offered senior product manager (the acquisition shaved a notch off of his former title of VP of Product Management) Jeff Fried as a substitute. All good.

Fortunately, I knew that I could trust my manager to live up to his commitments. I did have to fill out his online registration form on site at 7:30am, but he more than made up for it by buying me drinks after the conference. Two down, one to go.

Then there was Autonomy. Strangely, despite having working in this space for nearly a decade, I’d never actually met anyone from Autonomy. That meant I’d have to cold-call someone to have any chance of getting them to participate in the panel. Who to call? I decided that, since industry conference participation was probably under the umbrella of corporate marketing, I’d try their head of PR, one of the few Autonomy employees whose contact information is published on the open web.

Success! Or so I thought at the time. She told me that Peter Menell would participate in the panel, and, with my roster complete, I proceeded to start publicizing the Industry Track. However, a week and a half before the conference, she calls me to let me know that Peter can’t attend, and that in fact no one from Autonomy is available to take his place.

I was on vacation at the time, but of course conference organizers don’t get to take vacation, at least not when panelists cancel at the last minute. I started mulling over my options.

Including an empty chair on the panel would have given me the satisfaction of exposing Autonomy’s snub (both to me personally and to the information retrieval community), but I realized that wouldn’t be at all fair to the attendees.

Instead, I started going through the short list of possible panelists, favoring those who lived in the Boston area. And then, by fortuitous coincidence, I received an email from Raul Valdes-Perez, Executive Chairman and co-founder of Vivisimo: “If by any chance you need any last-minute help or stand-in, let me know.” I made a mental note to investigate Raul as a possible psychic and then happily welcomed him on board. And breathed a sigh of relief.

For all of the upheaval in bringing the panel together, the actual session went like clockwork. The panelists were respectful, disciplined, and yet not afraid to take risks in their stances. They talked about the challenges of holistically evaluating search applications, the interplay between relevance ranking and faceted search, the pros and cons of federated search, and much more. HCIR was a major theme, though perhaps that’s not surprising given the participants. As usual, Mary McKenna took more detailed notes.

I take pride both in the overall quality of the panel and specifically in the performance of my manager, whose background is more in systems and databases than in information retrieval. Of course, I’m biased, and he does sign my expense forms. 🙂

Regardless, I think I can muster enough objectivity to say that the panel was a huge success. Bruce Croft’s response, which also ended the Industry Track, was a fitting valedictory address: he urged us not to just walk away from the conversation. I am proud of the success of the Industry Track as an event, but I hope it is only the beginning of a deeper mingling of researchers and practitioners.

To James Allan and Jay Aslam, the SIGIR 2009 co-chairs, I thank you for the privilege and opportunity to organize the Industry Track. And, to whoever takes on such a responsibility at future SIGIR conferences, I hope you can benefit from my experience without having to relive it!

The morning sessions of the SIGIR 2009 Industry Track consisted of five individual presentations; the afternoon consisted of two panels. The requirement to synchronize with the research talks led to the allocation of 90 minutes for each panel–which was a bit more than I’d originally planned on (and this change, like many, occurred in the two weeks before the conference). James Allan, one of the SIGIR 2009 co-chairs, suggested that we add an academic responder to each of the panels to account for the additional time, and we went with that approach.

The first of the two afternoon panels consisted of industry analysts: Whit Andrews (Gartner), Sue Feldman (IDC), and Theresa Regli (CMS Watch). I moderated the panel–or, more accurately, attempted to moderate it. Marti Hearst served as the academic responder.

The panel opened with each of the three panelists making an opening statement, sharing their perspectives about the key business concerns and trends in the search industry. I asked them to talk about enterprise search in the broadest sense of the term–search applications that companies buy or build)–rather than in the narrow sense of no-frills intranet search.

It became immediately clear from their opening statements that the panelists had wildly different perspectives and styles. While I think the term “food fight” that I heard bandied around afterward is a bit of an exaggeration, they certainly engaged in a heated debate.

One topic that attracted particular controversy was how enterprise search applications should assign relevance to search results. Whit suggested that, in an enterprise setting, the main objective function is the profitability of the enterprise, and that relevance should essentially be money driven. Sue and Theresa disagreed sharply, mainly arguing that relevance should be user-controlled.

I’m probably oversimplifying their arguments, and in any case shouldn’t take sides in a debate among analysts! Still, Whit probably won’t be surprised that my sympathies generally lie with the users. That said, Whit is right that enterprise search companies sell to enterprises, not directly to customers, and those enterprises (like web search companies that sell to advertisers) may have interests that aren’t always aligned with those of users. At Endeca, we advise our customers on how to configure and communicate a relevance ranking strategy, but ultimately our customers make their own decisions. After all, it’s their site and their money.

And that leads to the other topic that caught my attention and came up during Marti’s responder session: the question of how analyst firms make money. All three of the panelists were open about how their employers make money, whether from enterprise buyers, vendors, or some combination thereof. My personal preference would be that analysts make money primarily from enterprise buyers–but of course I work for a frugal vendor. I’ve heard from a variety of sources that Endeca “doesn’t spend enough” on analyst services–or on corporate marketing in general. Since neither vendors nor analyst firms open up their books, I can only speculate. Fortunately, it’s clear the analysts on the panel not only have integrity, but also have strong enough views that they can’t probably couldn’t be swayed by money or pressure.

As organizer of the track, I inrended for the panel to offer an audience of mostly academic types a chance to see people whose opinions influence tens (if not hundreds) of millions of dollars in purchasing decisions. I hope I accomplished that. I’m especially grateful to these highly billable analysts for freely sharing their time and ideas. Neither SIGIR nor I could possibly have afforded their market rates!

For other perspectives, take a look at Theresa’s blog post, “Know Your Relevance“, or Mary McKenna’s summary post.

One of the things I didn’t consider when I signed on to organize the SIGIR 2009 Industry Track was that I’d have to replace speakers and panelists on less than two weeks’ notice. But what I couldn’t even have imagined was replacing a speaker on less than 24 hours’ notice!

Tuesday morning, the second day of the conference and the day before the Industry Track, I woke up to an email from Tip House of the OCLC, whom I’d planned to have speak about his experiences developing Worldcat.org, the world’s largest bibliographic database. Unfortunately, he had fallen ill and would not be able to make it to the conference.

I was determined not to have a hole in the program. I immediately sent an email to the Director of Search at LinkedIn, whom I had just met at the poster session the previous evening, hoping he might have a presentation tucked away about LinkedIn’s recent launch of faceted people search. I turned to Twitter–which actually earned me a plausible suggestion.

But it was during the morning coffee break that serendipity struck. As I walked by the Bing exhibitor table, I saw Jan Pedersen, Chief Scientist of Core Search at Microsoft, chatting with Peter Bailey, an applied researcher on the Bing team. I turned to them and, in my most charming voice, asked if they might be interested in having someone on their team talk about Bing the next day. They took a few minutes to think it over, and then replied in the affirmative, producing Nick Craswell, also an applied researcher. Problem solved, and I can proudly say that I Binged for it!

Nick talked about how query modeling, focusing issues like query ambiguity, session context, and temporal query dynamics (particularly seasonality). He talked a bit about a technique that involved random walks on click logs–a technique I remember striking me when I first heard him talk about it at ECIR 2008.

The talk was a bit raw–understandably so given the short notice. But it was great to see a major web search practitioner connecting information retrieval research to actual product. Yes, there were the standard caveats about not revealing secret sauce, but the talk was open and substantive. Indeed, I hope Nick will be able to share the slides!

UPDATE: Nick emailed me the slides and gave me permission to post them here.

Back to our regularly scheduled blogging about the SIGIR 2009 Industry Track. For those who haven’t been reading along, we covered the first three talks:

• Matt Cutts (Google): Web Spam and Adversarial IR: The Road Ahead
• danah boyd (Microsoft Research): The Searchable Nature of Acts in Networked Publics
• Vanja Josifovski (Yahoo! Research): Ad Retrieval – A New Frontier of Information Retrieval

As you can see, that covered the three major web search engine companies, at least at the time of the conference. Sure, danah’s talk wasn’t exactly what people might have expected, but I had some creative license as an organizer, and the audience loved her talk. Besides, as we’ll get to in the next post, Microsoft had other opportunities to present representatives of its more conventional information retrieval divisions.

The next speaker, according to the original plan, was to be Tom Tague, who leads the Open Calais project at Thomson Reuters. Unfortunately, a week before SIGIR, I found out that he would be unable to make it. One of his colleagues offered to present in his stead less than 24 hours after his cancellation, but by then I’d already found a replacement on my own: Evan Sandhaus, Semantic Technologist in the New York Times Research and Development Labs, agreed to talk about “Corpus Linguistics and Semantic Technology at the New York Times”.

You can get a good idea of his talk from these slides–or, better, yet, from this video. Both are from the closing keynote that he and his colleague Rob Larson delivered at the 2009 Semantic Technology Conference.

I won’t try to recapture Evan’s fascinating narrative about the history of information storage and retrieval at the New York Times. Rather, I’ll skip to the parts that should matter most to information retrieval researchers and practitioners: the availability of the New York Times Annotated Corpus through the Linguistic Data Consortium (LDC), and the New York Times’s intention to contribute to the Linked Data Cloud.

For me personally, the annotated corpus is the bigger deal. It represents 1.8 million articles written over 20 years. It is annotated both with manually-supplied summaries and tags–the latter drawn from a controlled vocabulary of people, organizations, locations and topic descriptors–and with algorithmically-supplied tags that are manually verified. My colleagues and I at Endeca have been working with the annotated corpus, and it is a delight. I hope that the information retriwval community will make heavy use of this wonderful new resource.

I’d hoped to get through all of the SIGIR 2009 Industry Track before blogging about anything else (such as Yahoo! search going bada-Bing), but clearly I’m taking too long. So I’m following Daniel Lemire’s suggestion that I post a recent comment on Lance Fortnow’s blog (actually a response to his CACM column entitled “Time for Computer Science to Grow Up“) here at The Noisy Channel.

It’s nice to see this piece joining a growing chorus questioning the way we conflate the distinct concerns of disseminating knowledge, establishing professional reputation, and building community. This problem is not unique to computer science, but we are certainly in a position to lead by example in addressing it.

In age where distribution is nearly free, I agree that we should move the filtering role from content publishers to content consumers. There’s no economic reason today why scholarship (or purported scholarship) shouldn’t be published online. Of course, the ability to publish digital content for free (or close to free) does not imply anyone will (or should) read what you write. The blogosphere offers an instructive example: the overwhelming majority of blogs attract few (if any) readers. I suspect that the same holds true for arXiv.org. Of course, peer-reviewed content may not fare that much better, particularly given the proliferation of peer-reviewed venues. Regardless, it makes no sense for publishers to act as filters in an age of nearly-free digital distribution.

That brings us to the question of how researchers should establish their professional reputation–and, in the case of academics, obtain tenure and promotion. Today, they have to publish in peer-reviewed journals and conferences. Even if we accept the weaknesses of the current peer-review regime, we should be able to separate content assessment from distribution. The peer-review process (and review processes in general) should serve to endorse content–and ideally even to improve it–rather than to filter it.

Finally, conferences should primarily serve to build community. I find the main value of conferences and workshops to be face-to-face interaction, and I’ve heard many people express similar sentiments. Part of the problem is that so few presenters at conferences invest in (or have the skills for) delivering strong presentations. But more fundamentally it’s not even clear that the presentations are the point of a conference–after all, an author’s main motive for submitting an article to a conference seems to be getting it into the proceedings.

Here are some questions I’d like to suggest we consider as a community:

What if presentation at a conference were optional, and an author’s decision to present had no effect on inclusion in the proceedings? Would there be significantly fewer presentations? Would those fewer presentation be of higher quality?

What if the process of peer-reviewing conference submissions required the submission of presentation materials rather than (or in addition to) a paper? Would the accepted presentations be of higher quality? Would researchers invest more in presentation skills? What would happen to strong researchers without such skills?

Can we update the traditional conference format to foster more productive interaction among researchers? For example, should we have more poster sessions and fewer paper presentations?

I’d love to see the computer science community take the lead in evolving what increasingly feel like dated procedures for disseminating knowledge, establishing professional reputation, and building community. I’ve tried to do my small part, co-organizing workshops on Human-Computer Interaction and Information Retrieval (HCIR) that emphasize face-to-face interaction and organizing the SIGIR 2009 Industry Track as a series of invited talks and panels from strong presenters. But I’m encouraged to see “establishment” types like Moshe Vardi and Lance Fortnow leading the charge to question the status quo.