With 70 registered attendees, it was the biggest HCIR workshop we have held. Rutgers was a gracious host, providing space not only for the all-day workshop but also for a welcome reception the night before.

And, based on an informal survey of participants, I can say with some semblance of objectivity that this was the best HCIR workshop to date.

The opening “poster boaster” session was particularly energetic. There was no award for best boaster, but Cathal Hoare won an ovation by delivering his boaster as a poem:

If a picture is worth a thousand words

Surely to query formulation a photo affords

The ability to ask ‘what is that’ in ways that are many

But for years we have asked how can-we

Narrow the search space so that in reasonable time

We can use images to answer questions that are yours and mine

In my humble poster I will describe

How recent technology and users prescribe

A solution that allows me to point and click

And get answers so that I don’t feel so thick

About my location and my environment

And to my touristic explorations bring some enjoyment

Now if after all that you feel rather dazed

Please come by my poster and see if you are amazed….

As in past years, we enlisted a rock-star keynote speaker–this time, Google UX researcher Dan Russell. His slides hardly do justice to his talk–especially without the audio and video–but I’ve embedded them here so that you can get a flavor for his presentation on how we need to do more to improve the searcher.

We accepted six papers for the presentation sessions–sadly, one of the presenters could not make it because of visa issues. The five presentations covered a variety of topics relating to tools, models, and evaluation for HCIR. The most intriguing of these (to me, at least) was a presentation by Max Wilson about “casual-leisure searching”–which he argues breaks our current models of exploratory search. Check out the slides below, as well as Erica Naone’s article in Technology Review on “Searching for Fun“.

I’ve been exchanging emails with Dhiti co-founder Bharath Mohan about Nuggetize, an intriguing interface that surfaces “nuggets” from a site to reduce the user’s cost of exploring a document collection. Specifically Nuggetize targets research scenarios where users are likely to assemble a substantial reading list before diving into it. You can try Nuggetize on the general web or on a particular site that has been “nuggetized”, e.g., a blog like this one or Chris Dixon’s.

I’m always happy to see people building systems that explicitly support exploratory search (and am looking forward to seeing the HCIR Challenge entries in a week!). Regular readers may recall my coverage of Cuil, Kosmix, and Duck Duck Go. And of course I helped build a few of my own at Endeca. So what’s special about Nuggetize?

Mohan describes it as a faceted search interface for the web. I’ll quibble here–the interface offers grouped refinement options, but the groups don’t really strike me as facets. Moreover, the interface isn’t really designed to explore intersections of the refinement options–rather, at any given time, you see the intersection of the initial search and a currently selected refinement. But it is certainly an interface that supports query refinement and exploration.

The more interesting features are the nuggets and the support for relevance feedback.

The nuggets are full sentences, and thus feel quite different from conventional search-engine snippets. Conventional snippets serve primarily to provide information scent, helping users quickly determine the utility of a search result without the cost of clicking through to it and reading it. In contrast the nuggets are document fragments that are sufficiently self-contained to communicate a coherent thought. The experience suggests passage retrieval rather than document retrieval.

The relevance feedback is explicit: users can thumbs-up or thumbs-down results. After supplying feedback, users can refresh their results (which re-ranks them) and are also presented with suggested categories to use for feedback (both positive and negative). Unfortunately, the research on relevance feedback tells us that, helpful as it could be to improving user experience, users don’t bite. But perhaps users in research scenarios will give it a chance–especially with the added expressiveness and transparency of combining document and category feedback.

Overall it is a slick interface, and it’s nice seeing the various ideas Mohan and his colleagues put together. There’s certainly room for improvement–particularly in the quality of the categories, which sometimes feel like victims of polysemy. Open-domain information extraction is hard! Some would even call it a grand challenge.

Mohan reads this blog (he reached out to me a few months ago via a comment), and I’m sure he’d be happy to answer questions here.

If you’re a search engine junkie like me, you’ve probably heard about Blekko, a search engine that has been percolating for over two years and recently launched a private beta. If not, I encourage you to watch the TechCrunch video I’ve embedded above. You can join the beta by following them on Twitter. I did that earlier this week, and my invitation arrived via a direct message the next day.

Blekko’s main differentiating feature is that it supports “slashtags”. These aren’t the same as the Twitter microsyntax proposed by Chris Messina and named by Chris Blow. Rather, they are a way for users to “spin” their search results using a variety of filters. For example, [climate /liberal] and [climate /conservative] return very different results, because they are restricted to different sets of sites.

In addition to providing a set of curated slashtags, Blekko allows users to define their own slashtags by specifying the sets of sites to be included. There’s a social aspect here too: you can use (and follow) other users’ slashtags. Blekko also has some special slashtags that don’t act as site filters, e.g., /date shows recent results and /seo offers indexing information about web sites.

Blekko emphasizes two characteristics that I find very appealing: transparency and user control. While they do not disclose their relevance ranking algorithm, they do expose some of the information they use to compute it. More significantly, their emphasis on slashtags de-emphasizes default ranking, but rather encourages users to take more responsibility in the information seeking process. Very HCIR!

I like the concept. But I’m not sure how I feel about the execution. I have three main concerns.

First, the set of slashtags is somewhat haphazard–to be expected in a beta, but I’m not sure how it will evolve. I’d love to see a vocabulary collectively (and transparently) curated like Wikipedia, but I fear it will look more like social tagging site Delicious, which is a case study in the “vocabulary problem“. As any information scientist can tell you, managing vocabularies is hard!

Second, I’m not sure if site filters are the right model. What happens to sites with heterogeneous content? Or to sites that have one-hit wonders and therefore are unlikely to show up in any slashtags? I’d prefer to see the sites used as seeds to train classifiers that could then be applied to the entire index. Something a bit more like what Miles Efron implemented in this research–only on a much larger scale and applied at a page rather than site level.

Third, I think there’s a third ingredient that is essential to complement transparency and user control: guidance. As a user, I need to know what slashtags would lead me to interesting results, and ideally I’d want some kind of preview to make exploration as low-cost as possible.

I know I’m asking for a lot–especially from an ambitious startup that has just launched its private beta. But I think the stakes are high in this space, and going easy on a newcomer is no favor. I offer the tough love of a critic who would really like to see this kind of vision succeed.

If you are are interested in attending and haven’t already registered, please let me know as soon as possible via email or Twitter (speaking of which, follow the #hcir2010 hash tag). We’re making the remaining slots available to the community on a first-come, first-serve basis.

Google user experience researcher Dan Russell will be delivering this year’s keynote on “Why is search sometimes easy and sometimes hard? Understanding serendipity and expertise in the mind of the searcher“.

Here is the list of accepted papers:

Oral Presentations

• VISTO: for Web Information Gathering and Organization
  Anwar Alhenshiri, Carolyn Watters, and Michael Shepherd (Dalhousie University)
• Time-based Exploration of News Archives
  Omar Alonso (Microsoft Corporation), Klaus Berberich (Max-Planck Institute for Informatics), Srikanta Bedathur (Max-Planck Institute for Informatics), and Gerhard Weikum (Max-Planck Institute for Informatics)
• Combining Computational Analyses and Interactive Visualization to Enhance Information Retrieval
  Carsten Goerg, Jaeyeon Kihm, Jaegul Choo, Zhicheng Liu, Sivasailam Muthiah, Haesun Park, and John Stasko (Georgia Institute of Technology)
• Impact of Retrieval Precision on Perceived Difficulty and Other User Measures
  Mark Smucker and Chandra Prakash Jethani (University of Waterloo)
• Exploratory Searching As Conceptual Exploration
  Pertti Vakkari (University of Tampere)
• Casual-leisure Searching: The Exploratory Search Scenarios that Break our Current Models
  Max L. Wilson (Swansea University) and David Elsweiler (University of Erlangen)

HCIR Challenge Reports

• Search for Journalists: New York Times Challenge Report
  Corrado Boscarino, Arjen P. de Vries, and Wouter Alink (Centrum Wiskunde and Informatica)
• Exploring the New York Times Corpus with NewsClub
  Christian Kohlschütter (Leibniz Universität Hannover)
• Searching Through Time in the New York Times
  Michael Matthews, Pancho Tolchinsky, Roi Blanco, Jordi Atserias, Peter Mika, and Hugo Zaragoza (Yahoo! Labs)
• News Sync: Three Reasons to Visualize News Better
  V.G. Vinod Vydiswaran (University of Illinois), Jeroen van den Eijkhof (University of Washington), Raman Chandrasekar (Microsoft Research), Ann Paradiso (Microsoft Research), and Jim St. George (Microsoft Research)
• Custom Dimensions for Text Corpus Navigation
  Vladimir Zelevinsky (Endeca Technologies)
• A Retrieval System Based on Sentiment Analysis
  Wei Zheng and Hui Fang (University of Delaware)

Research Posters

• Improving Web Search for Information Gathering: Visualization in Effect
  Anwar Alhenshiri, Carolyn Watters, and Michael Shepherd (Dalhousie University)
• User-oriented and Eye-Tracking-based Evaluation of an Interactive Search System
  Thomas Beckers and Norbert Fuhr (University of Duisberg-Essen)
• Exploring Combinations of Sources for Interaction Features for Document Re-ranking
  Emanuele Di Buccio (University of Padua), Massimo Melucci (University of Padua), and Dawei Song (The Robert Gordon University)
• Extracting Expertise to Facilitate Exploratory Search and Information Discovery: Combining Information Retrieval Techniques with a Computational Cognitive Model
  Wai-Tat Fu and Wei Dong (University of Illinois at Urbana-Champaign)
• An Architecture for Real-time Textual Query Term Extraction from Images
  Cathal Hoare and Humphrey Sorensen (University College Cork)
• Transaction Log Analysis of User Actions in a Faceted Library Catalog Interface
  Bill Kules (The Catholic University of America), Robert Capra (University of North Carolina at Chapel Hill), and Joseph Ryan (North Carolina State University Libraries)
• Context in Health Information Retrieval: What and Where
  Carla Lopes and Cristina Ribeiro (University of Porto)
• Tactics for Information Search in a Public and an Academic Library Catalog with Faceted Interfaces
  Xi Niu and Bradley M. Hemminger (University of North Carolina at Chapel Hill)

Position Papers

• Understanding Information Seeking in the Patent Domain and its Impact on the Interface Design of IR Systems
  Daniela Becks, Matthias Görtz, and Christa Womser-Hacker (University of Hildesheim)
• Better Search Applications Through Domain Specific Context Descriptions
  Corrado Boscarino, Arjen P. de Vries, and Jacco van Ossenbruggen (Centrum Wiskunde and Informatica)
• Layered, Adaptive Results: Interaction Concepts for Large, Heterogeneous Data Sets
  Duane Degler (Design for Context)
• Revisiting Exploratory Search from the HCI Perspective
  Abdigani Diriye (University College London), Max L. Wilson (Swansea University), Ann Blandford (University College London), and Anastasios Tombros (Queen Mary University London)
• Supporting Task with Information Appliances: Taxonomy of Needs
  Sarah Gilbert, Lori McCay-Peet, and Elaine Toms (Dalhousie University)
• A Proposal for Measuring and Implementing Group’s Affective Relevance in Collaborative Information Seeking
  Roberto González-Ibáñez and Chirag Shah (Rutgers University)
• Evaluation of Music Information Retrieval: Towards a User-Centered Approach
  Xiao Hu (University of Illinois at Urbana Champaign) and Jingjing Liu (Rutgers University)
• Information Derivatives: A New Way to Examine Information Propagation
  Chirag Shah (Rutgers University)
• Implicit Factors in Networked Information Feeds
  Fred Stutzman (University of North Carolina at Chapel Hill)
• Improving the Online News Experience
  V. G. Vinod Vydiswaran (University of Illinois) and Raman Chandrasekar (Microsoft Research)
• Breaking Down the Assumptions of Faceted Search
  Vladimir Zelevinsky (Endeca Technologies)
• A Survey of User Interfaces in Content-based Image Search Engines on the Web
  Danyang Zhang (The City University of New York)

You can also download the full proceedings here.

So there’s no question that community question answering is hot. The question is why? In particular, is community question answering a step forward or backward relative to today’s search engines, or is it something different?

Regarding Facebook Questions, Jason Kincaid writes in TechCrunch:

Given its size, it won’t take long for Facebook to build up a massive amount of data — if that data is consistently reliable, Questions could turn into a viable alternative to Google for many queries.

That’s a big if.  But I think the bigger caveat is the vague quantifier “many”. The success of community question answering services will depend on how these services position themselves relative to users’ information needs. Anyone arguing that these services can or should replace today’s web search engines might want to consider the following examples of information needs that are typical of current search engine use:

• How do I get an iPhone case?
• Who sings the “choco latte” song?
• What movies are playing in my neighborhood?
• How do I get to Boston from New York?
• What is the best selling netbook?
• Who offers the best cell phone reception in New York?
• What was the score in the North Korea – Portugal game?

I hope I don’t have to keep going to convince you that web search engines have earned their popularity by serving a broad class of information needs (i.e., answer lots of questions)–and that’s without even using the wide variety of personalized and social features that web search engines are rapidly developing.

The common thread in the above questions is that they focus on objective information. In general, such questions are effectively and efficiently answered by search engines based on indexed, published content (including “deep web” content made available to search engines via APIs). There’s a lot of work we can do to improve search engines, particularly in the area of supporting query formulation. But it seems silly and wasteful to route such questions to other people–human beings should not be reduced to performing tasks at which machines excel.

That said, I agree with Kincaid that there are many information needs that are well addressed by  community question answering. In particular:

• Questions for which point of view is a feature, not a bug. Review sites succeed when they provide sincere, informed personal reactions to products and services. Similarly, routing questions to people makes sense either when we care about the answerer’s a point of view. For some questions, I want the opinion of someone who shares my taste (which is what Hunch is pursuing with its “taste graph“). For others, I want a diversity of expert opinions–for which I might turn to Aardvark (which tries to route questions to topic experts), Quora (where people follow particular topics), or LinkedIn Answers. Over time, the answers to many such questions can be published and indexed–and indeed some answers sites receive a large share of their traffic from search engines.
• Niche topics. As much as web search as improved information accessibility for the “long tail” of published information, the effectiveness of web search can be highly variable for the most obscure information needs. Moreover, this effectiveness depends significantly on the user: some people are better at searching than others, especially in their areas of domain expertise. Social search can help level the playing field. Much as Wikipedia has surfaced much of the expertise at the head of the information distribution, community question answering can help out in the tail.
• Community for its own sake. Even in cases where search engines are more effective and efficient than community question answering services, some people prefer to participate in a social exchange rather than to conduct a transaction with an impersonal algorithm. Indeed, researchers at Aardvark found that many of the questions posed through their service (pre-acquisition) could be answered successfully using Google. I’ll go out on a limb and assume that Aardvark’s users were early technology adopters who are quite conversant with search engines–but in some case chose to use a social alternative simply because they wanted to be social.

Conclusions? Community question answering may be overhyped right now, but it isn’t a fad. There are broad classes of subjective information needs that require a point of view, if not a diversity of views. And even if much of the use of community question answering sites is mediated by search engines indexing their archives, there will always be a need for fresh content. I also believe that social search will continue to be valuable for niche topics, since neither search engines nor searchers will ever be perfect.

But I think the biggest open question is whether people will favor community question answering simply to be social. I conjecture that, by very publicly integrating community question answering into is social networking platform, Facebook is testing the hypothesis that it can turn information seeking from a utilitarian individual task into an entertaining social destination. Given Facebook’s highly engaged user population, we won’t have to wait long to find out.

• The new frontiers of Web search: going beyond the 10 blue links
  Ricardo Baeza-Yates, Andrei Broder, Yoelle Maarek, and Prabhakar Raghavan, Yahoo! Labs
  
• Cross-Language Information Retrieval in the Legal Domain
  Samir Abdou and Thomas Arni, Eurospider
• Building and Configuring a Real-Time Indexing System
  Garret Swart, Ravi Palakodety, Mohammad Faisal, Wesley Lin, Oracle
  
• Lessons and Challenges from Product Search
  Daniel E. Rose, A9.com (Amazon)
  
• Being Social: Research in Context-aware and Personalized Information Access @ Telefonica
  Xavier Amatriain, Karen Church and Josep M. Pujol, Telefónica
  
• Searching and Finding in a Long Tail Marketplace
  Neel Sundaresan, eBay
  
• When No Clicks are Good News
  Carlos Castillo, Aris Gionis, Ronny Lempel, and Yoelle Maarek, Yahoo! Research

I missed the Eurospider and Oracle talks, but otherwise I spent the afternoon enjoying these sessions. The slides, along with all of the keynote slides, are available here.

Some highlights from the talks I attended:

• Andrei Broder, a pioneer of Web IR and author of the highly cited “Taxonomy of Web Search“,  enumerated a half-dozen challenges for web search to move from its current state to one that not only accomplishes semantic analysis but also supports task completion. Naturally, the one that appeals to me is the need for search engines to move beyond query suggestion and truly engage the user in a dialog.
• Dan Rose talked about the challenges of product search, and in particular the blessing and curse of implementing search applications for structured data (something that I’m very familiar with from my previous role at Endeca). He also warned of the dangers of over-interpreting behavioral data, e.g., a site change that increases revenue does not necessarily imply a better user experience (it could just be favoring higher-priced inventory), and may ultimately alienate customers.
• Xavier Amatriain focused on social search, and talked about how, as we’ve turned to context to help mitigate information overload, we find ourselves confronted with the new problem of context overload. Specifically, he cited the research questioning the wisdom of the crowd, and proposed the wisdom of the (expert) few as a better alternative.
• Neel Sundaresan offered an interesting tour of eBay Research Labs prototypes, including the BayEstimate that helps sellers improve listing titles by discovering the keywords are both representative of the item and used in buyers’ queries.
• Finally, Carlos Castillo offered a nice approach to discover when search engine abandonment is “good abandonment“: identify a subset of “tenacious” users who almost never abandon searches and measure their abandonment–since it is almost certain to be the good kind.

All in all, I was very impressed with the quality of the Industry Track, and gratified to see how it had improved on the program I put together last year. Given the key role that industry plays in information retrieval, I think it is important that the top-tier IR conference promote the best that industry has to offer.

But this year the SIGIR 2010 Industry Track broke new ground. The keynotes were from some of the most senior technologists at the world’s largest web search engines:

• William Chang, Chief Scientist at Baidu
• Yossi Matias, Head of Google’s Israel R&D Center
• Jan Pedersen, Chief Scientist for Core Search at Microsoft (Bing)
• Ilya Segalovich, CTO and Co-Founder of Yandex

I won’t attempt to provide much detail about these presentations, first because I’m hoping they will all be they have all been posted online and second because Jeff Dalton has already done an excellent job of posting live-blogged notes. Rather, I’ll offer a few reactions.

William’s presentation on the “Future Search: From Information Retrieval to Information Enabled Commerce” unsurprisingly focused on the Chinese search-related market. While the topic of  Google in China was an elephant in the room, it did not surface even obliquely in the presentation–and I commend William for taking the high road. As for Baidu itself, its most interesting innovation is Aladdin, an open search platform that allows participating webmasters to submit query-content pairs.

Yossi’s presentation on “Search Flavours at Google” was a tour de force of Google’s recent innovations in the search and data mining space. The search examples most focused on the challenges of incorporating context into query understanding–where context might involve geography, time, social network, etc. But some of the more impressive examples showed off using the power of data to predict the present. More than anything, his presentation made clear that Google is doing a lot more than returning the traditional ten blue links.

Jan talked about “Query Understanding at Bing”. I really hope he makes these slides available, since they do a really nice job of describing a machine learning based architecture for processing search queries. To get an idea of this topic, check out Nick Craswell’s presentation from last year’s SIGIR.

Finally Ilya talked about “Machine Learning in Search Quality at Yandex”, the largest search engine in Russia. He described the main challenge in Russia as handling the local aspects of search: he gave as an example that, if you’re in a small town in Russia, then local results in Moscow may as well be on the moon. Local search is a topic close to my heart, not least of which because it is my day job! Ilya’s talked focused largely on Yandex’s MatrixNet implementation of learning to rank. What I’m surprised he didn’t mention is the challenges of data acquisition–in general, for domains beyond the web, obtaining high-quality data is often a much bigger challenge than filtering and ranking it.

All in all, the four keynotes collectively offered an excellent state-of-the-search-engine address.  As with last year, the industry track talks were the most popular morning sessions, and the speakers delivered the goods.

In the morning session, I attended three talks. The first, “Geometric Representations for Multiple Documents” by Jangwon Seo and Bruce Croft, looks at the problem of representing combinations of documents in a query model. It provides both theoretical and experimental evidence that geometric means work better than arithmetic means for representing such combinations. The second, “Using Statistical Decision Theory and Relevance Models for Query-Performance Prediction” by Anna Shtok, Oren Kurland, and David Carmel, shows the efficacy of a utility estimation framework comprised of relevance models, measures like query clarity to estimate the representativeness of relevance models, and similarity measures to estimate the similarity or correlation between two ranked lists. The authors demonstrated significant improvements from the framework over simply using the representativeness measures for performance prediction. The third paper, “Evaluating Verbose Query Processing Techniques” by Samuel Huston and Bruce Croft, showed that removing “stop structures”, a generalization of stop words, could significantly improve performance on long queries. Interestingly, the authors evaluated their approach on “black box” commercial search engines Yahoo and Bing without knowledge of their retrieval models.

In the session after lunch, I mostly attended talks from the session on user feedback and user models. The first, “Incorporating Post-Click Behaviors Into a Click Model” by Feimin Zhong, Dong Wang, Gang Wang, Weizhu Chen, Yuchen Zhang, Zheng Chen, and Haixun Wang, proposed and experimentally validated a click model to infer document relevance from post-click behavior like dwell time that can be derived from logs. The second, “Interactive Retrieval Based on Faceted Feedback” by Lanbo Zhang and Yi Zhang, described an approach using facet values for relevance and pseudo-relevance feedback. It’s interesting work, but I think the authors should look at work my colleagues and I presented at HCIR 2008 on distinguishing whether facet values are useful for summarization or for refinement. The third, “Understanding Web Browsing Behaviors through Weibull Analysis of Dwell Time” by Chao Liu, Ryen White, and Susan Dumais, offered an elegant model of dwell time and used it to predict dwell time distribution from page-level features. Finally, I attended one talk from the session on retrieval models and ranking: “Finding Support Sentences for Entities” by Roi Blanco and Hugo Zaragoza. They present a novel approach of generalizing snippets to interfaces that offer named entities (e.g., people) as supplements to the search results. I am excited to see research that could make richer interfaces more explainable to users.

I spend the last session of the day listening to a couple of talks about users and interactive IR. The first was “Studying Trailfinding Algorithms for Enhanced Web Search” by Adish Singla, Ryen White, and Jeff Huang, turned out to be the best-paper winner. This work extends previous work that Ryen and colleagues have done on search trails and showed results of various trailfinding algorithms that outperform the trails users follow on their own. The second, “Context-Aware Ranking in Web Search” by Biao Xiang, Daxin Jiang, Jian Pei, Xiaohui Sun, Enhong Chen, and Hang Li, analyzes requerying behavior as reformulation, specialization, generalization, or general association, and demonstrates that knowing or inferring which the user is doing significantly improves ranking of the second query’s results.

The day wrapped up with a luxurious banquet at the Hotel Intercontinental, near the Nations Plaza. After sweating through conference sessions without air conditioning, it was a welcome surprise to enjoy great food in such an elegant setting.

But of course she did a lot more than defend Cranfield. She offered a comprehensive and fascinating history of the Cranfield paradigm, starting with the Cranfield 1 experiments in the late 1950s which evaluated manual indexing systems.

Most importantly, she defined the Cranfield paradigm as defining a metric that reflects real user model and building the collection before the experiments to prevent human bias and enable reusability. As she noted, this model does not say anything about only returning a ranked list of ten blue links–which is what most people (myself included) associate with the Cranfield model. Indeed, she urged us to think outside this mindset.

I loved the presentation and found the history enlightening (though Stephen Robertson corrected a few minor details). Still, I wondered if she was defining the Cranfield paradigm so broadly as to co-opt all of its critics. But I think the clear dividing line between Cranfield and non-Cranfield is whether user effects are something to avoid or embrace. I perceive the success of Cranfield as coming in large part from its reduction of user effects. But I think that much of the HCIR community sees user effects as precisely what we need to be evaluating for information seeking support systems.

In any case, it was a great keynote, and Donna promises me she will make the slides available. Of course I’ll post them here. In the mean time, check out Jeff Dalton’s notes on his great blog and the tweets at #sigir2010.

• “Query Quality: User Ratings and System Predictions” by Claudia Hauff, Franciska de Jong, Diane Kelly, and Leif Azzopardi offered the startling (to me at least) result that human prediction of query difficulty did not correlate (or at best correlated weakly) to post-retrieval query performance prediction (QPP) measures like query clarity. I talked with Diane about it, and I wonder how strongly the human prediction, which was pre-retrieval, would correlate to human assessments of the results. I also don’t know how well the QPP measures she used apply to web search contexts.
• Which leads me to the next poster I saw, “Predicting Query Performance on the Web” by Niranjan Balasubramanian, Giridhar Kumaran, and Vitor Carvalho. They offered what I saw as a much more encouraging result–namely that QPP is highly reliable when it returns low scores. In other words, a search engine may wrongly believe that it did well on a query, but it is almost certainly right when it thinks it failed. This certainty on the negative side is exactly the opening that HCIR advocates need to offer richer interaction for queries a conventional ranking approach recognizes its own failure. While some of the specifics of the authors’ approach are proprietary (they perform regression on features used by Bing), the approach seems broadly applicable.
• Next I saw “Hashtag Retrieval in a Microblogging Environment” by Miles Efron. He provided evidence that hashtags could be an effective foundation for query expansion of Twitter search queries, using a language model approach. The approach may generalize beyond hashtags, but hashtags do have the advantage of being highly topical and relatively unambiguous by convention.
• “The Power of Naive Query Segmentation” by Matthias Hagen, Martin Potthast, Benno Stein, and Christof Brautigam suggested a simple approach for segmenting long queries into quoted phrases: consider all segmentations and, for a given segmentation, compute a weighted sum of the Google ngram counts for each quoted phases, the weight of a phrase of length s being s^s. I don’t find the weighting particularly intuitive, but the accuracy numbers they present look quite nice relative to more sophisticated approaches.
• “Investigating the Suboptimality and Instability of Pseudo-Relevance Feedback” by Raghavendra Udupa and Abhijit Bhole showed that an oracle with knowledge of a few high-scoring non-relevant documents could vastly improve the performance of pseudo-relevance feedback. While this information does not lead directly to any applications, it does suggest that obtaining a very small amount of feedback from the user might go a long way. I’m curious how much is possible from even a single negative-feedback input.
• “Short Text Classification in Twitter to Improve Information Seeking” by Bharath Sriram, David Fuhry, Engin Demir, Hakan Ferhatosmanoglu, and Murat Demirbas challenged the conventional wisdom that tweets are too short for traditional classification methods. They achieved nice results, but on the relatively simple problem of classifying tweets as news, events, opinions, deals, and private messages. I was offered promises of future work, but I think the more general classification problem is much harder.
• “Metrics for Assessing Sets of Subtopics” by Filip Radlinski, Martin Szummer, and Nick Craswell proposed an evaluation framework for result diversity based on coherence, distinctness, plausibility, and completeness. I suggested that this framework would apply nicely to faceted search interfaces, and that I’d love to see it demonstrated on production systems–especially since I think that might be easier to achieve than convincing the SIGIR community to embrace it.
• Which leads me nicely to the last poster I saw, “Machine Learned Ranking of Entity Facets” by Roelof van Zwol, Lluis Garcia Pueyo, Mridul Muralidharan, and Borkur Sigurbjornsson. They found that they could accurately predict click-through rates on named entity facets (people, places) by learning from click logs. It’s worth noting that their entity facets are extremely clean, since they are derived from sources like Wikipedia, IMDB, GeoPlanet, and Freebase. It’s not clear to me how well their approach would work for noisier facets extracted from open-domain data.

As I said, there were over a hundred posters, and I’d meant to see far more of them. Hopefully other people will blog about some of them! Or perhaps tweet about them at #sigir2010.

Jinyoung Kim of UMass presented joint work with Bruce Croft on “Ranking using Multiple Document Types in Desktop Search” in which they showed that type prediction can significantly improve known-item search performance in simulated desktop settings. I like the approach and result, but I’d be very interested to see how well it applied to more recall-oriented tasks.

Then came work by Googlers Enrique Alfonseca, Marius Pasca, and Enrique Robledo-Arnuncio on “Acquisition of Instance Attributes via Labeled and Related Instances” that overcomes the data sparseness of open-domain attribute extraction by computing relationships among instances and injecting this relatedness data into the instance-attribute graph so that attributes can be propagated to more instances. This is a nice enhancement to earlier work by Pasca and others on obtaining these instance-attribute graphs.

The session ended with an intriguing paper on “Relevance and Ranking in Online Dating Systems” by Yahoo researchers Fernando Diaz, Donald Metzler, and Sihem Amer-Yahia that formulated a two-way relevance model for matchmaking systems but unfortunately found that it did no better than query-independent ranking in the context of a production personals system. I would be very interested to see how the model applied to other matchmaking scenarios, such as matching job seekers to employers.

After a wonderful lunch hosted by Morgan & Claypool for authors, I attended a session on Filtering and Recommendation.

It started with a paper on “Social Media Recommendation Based on People and Tags” by IBM researchers Ido Guy, Naama Zwerdling, Inbal Ronen, David Carmel, and Erel Uziel. They analyzed item recommendation in an enterprise setting and found that a hybrid approach combining algorithmic tag-based recommendations with people-based recommendations achieves better performance at delivering interesting recommendations than either approach alone. I’m curious how well these results generalize outside of enterprise settings–or even how well they apply across the large variation in enterprises.

Then came work by Nikolaos Nanas, Manolis Vavalis, and Anne De Roeck on “A Network-Based Model for High-Dimensional Information Filtering”. The authors propose to overcome the “curse of dimensionality” of vector space representations of profiles by instead modeling keyword dependencies in a directed graph and applying a non-iterative activation model to it. The presentation was excellent, but I’m not entirely convinced by the baseline they used for their comparisons.

After that was a paper by Neal Lathia, Stephen Halles, Licia Capra, and Xavier Amatriain on “Temporal Diversity in Recommender Systems”. They focused on the problem that users get bored and frustrated by recommender systems that keep recommending the same items over time. They provided evidence that users prefer temporal diversity of recommendations and suggested some methods to promote it. I like the research, but I still think that recommendation engines cry out for transparency, and that transparency can also help address the diversity problem–e.g., pick a random movie the user watched and propose recommendations explicitly based on that movie.

Unfortunately I missed the last paper of the session, in which Noriaki Kawamae talked about “Serendipitous Recommendations via Innovators”.

Reminder: also check out the tweet stream with hash tag #sigir2010.

I arrived here on Monday, too jet-lagged to even imagine attending the tutorials, but fortunately I recovered enough to go to the welcome reception in the Parc de Bastions that evening. Then a night of sleep and on to the main event.

Tuesday morning kicked off with a keynote by Microsoft Live Labs director Gary Flake entitled “Zoomable UIs, Information Retrieval, and the Uncanny Valley”. Flake’s premise is that information retrieval is stuck in the “uncanny valley“, a metaphor he borrows from the robotics community. According to Wikipedia:

The theory holds that when robots and other facsimiles of humans look and act almost like actual humans, it causes a response of revulsion among human observers. The “valley” in question is a dip in a proposed graph of the positivity of human reaction as a function of a robot’s lifelikeness.

Flake offered Grokker (R.I.P.) as an example of a search interface that emphasized visual clustering and got stuck in the uncanny valley. He called it “the sexiest search experience that no one was going to use”. Flake then went on to propose that moving beyond the uncanny valley would require replacing our current discrete interactions with search engines into a mode of continuous, fluid interaction where whole of data greater than sum or parts. He offered some demos, emphasizing the recently released Pivot client, that he felt provided a vision to overcome the uncanny valley.

As became clear in the question and answer period, many people (myself included) felt that this rich visual approach might work well for browsing images but not as clear a fit for text-oriented information needs–despite Flake offering a demo based on the collection of Wikipedia documents. In fairness, it may be too early to assess a proof of concept.

Subject: we just got Beatles Rock Band for the office and are looking for a vocalist !!

Good Afternoon,

I hope you don’t mind me reaching out to you, but came across your LinkedIn page and my interest is peaked, to say the least. I hope after reading this you feel the same.

If you’re unfamiliar with XXXXXX, we are a distinct small and agile team that functions as an incubated start-up funded by a larger organization. What we are working on is still kind of a secret but I can tell you that it’s focused on completely changing the way we find, consume, share, and manage content on the web today. We are focused on the growing importance of the real-time web and the concurrent need to reduce the noise. We are driven by a strong desire to deliver a better overall experience with a lot less effort required from our users.

Our office is extremely open and collegiate, and we are committed to letting ideas thrive above all else. We’re a very eclectic bunch of characters, but we all share a common commitment to taking whatever we do, fun or work, to the max. Some words that have been used to describe us are: passionate, fun, funny, innovative, contrarian, automagical, brilliant, academic, whimsical, and most importantly respectful. If you fit 3 or more of those descriptions, you might just have some of that magic we’re looking for.

If you’re interested in exploring this opportunity, please email me your resume and I’ll follow up with you ASAP, and have you come by meet the team some time soon.

Either way, I hope to hear from you!

Have a great weekend,

We too are BIG karaoke fans ( I read your website) , and as I said above we just got Beatles Rock Band for the office and are looking for a vocalist !!

Cheers,
XXXXX

I see this email is a poster child of how a startup should recruit. It’s well-written, funny, and shares enough about the opportunity to be an effective hook. Most importantly it’s *personal*. Starting from the subject line that made a great first impression, the email showed proof that the sender–a complete stranger–had taken time to get to know about me.

This is a strategy that does not scale arbitrarily–and that is the whole point. A startup that is building a small team needs to choose its prospective employees carefully and then go after those prospects with full force. If you really want to earn someone’s attention, you have to show that you’ve invested attention yourself. There’s no free lunch–if you want to send out a hundred emails like this one, you’ve got your work cut out for you! But no startup should be recruiting on such a massive scale, and the increase in yield justifies the additional per-candidate investment.

Of course, this principle applies beyond the narrow context of recruiting. Indeed, it is much like an attention bond mechanism: prove to me that you’ve invested in targeting me personally, and I’ll be more inclined to invest my attention in reading your message. Indeed, search advertising follows a similar principle. I still maintain that search is not advertising, but perhaps this aspect of negotiating a shared interest between messenger and messengee is a common thread.

Hit songs, books, and movies are many times more successful than average, suggesting that “the best” alternatives are qualitatively different from “the rest”; yet experts routinely fail to predict which products will succeed. We investigated this paradox experimentally, by creating an artificial “music market” in which 14,341 participants downloaded previously unknown songs either with or without knowledge of previous participants’ choices. Increasing the strength of social influence increased both inequality and unpredictability of success. Success was also only partly determined by quality: The best songs rarely did poorly, and the worst rarely did well, but any other result was possible.

The result is hardly surprising to anyone familiar with the history of pop music. But I’m intrigued by the possibility that technology is simultaneously pulling music as a social phenomenon in two opposite directions.

On one hand, YouTube and social networks may actually be amplifying the positive feedback of music popularity. The recent story of YouTube sensation Greyson Chance (yes, a 13-year old with his own Wikipedia entry) becoming a national phenomenon in a couple of weeks attests to the power of social contagion. I don’t mean to take anything away from Chance’s talent, but I feel safe asserting that his talent was necessary but hardly sufficient to achieve his popular success.

On the other hand, Internet radio services like Pandora and Last.fm, despite their social features, offer the possibility of drastically reducing the effect of social influence. Both of these services require users to provide some representation of their musical tastes as initial inputs, whether by selecting preset stations or using particular artists or songs as seeds. Presumably those tastes are in large part the product of social influence. But the subsequent interaction between users and these services is relatively buffered from social influence. Users hear songs while listening privately through headphones–in many cases at work or while commuting. No one else is around when those users decide how to rate what they are listening to.

Granted, social context will always seep in–I don’t think I could give a thumbs-up to a Justin Bieber song even in the privacy of my own Pandora profile. But much of the music I discover is from artists I’ve never heard of–and thus evaluate without the explicit social influence of preconceptions about those artists.

As it turns out, I often discover after the fact that a number of the artists I like have achieved popular success. I can’t tell whether that reflects on their objective music quality, my own conformity of musical taste, or skew on the part of the recommendation system (cf. does everything sounds like Coldplay?). Still, I’m quite sure that I’m not favoring music based on prior knowledge of its popularity –for the most part, I don’t have that information at the time that I decide whether I like a song. Indeed, I hear new music almost exclusively through Pandora.

I don’t know how exceptional I am as a media consumer, but I suspect my case is increasingly common. Perhaps we are heading into a world where there will be a split between musical taste as social currency vs. musical taste as purely personal pleasure. It’s harder for me to imagine books or feature-length movies becoming so divorced from social context, if only because consuming them is a much larger and concentrated investment.

Still, I think it’s a big deal that this is happening in music. It’s a welcome counterpoint to the winner-take-all dynamic that has dominated the past decades of pop music. I can’t say that it will make the music industry more of a meritocracy–or that I even know what that would mean. But I think it’s a welcome step away from the caricature of conformity demonstrated by Salganik’s research.

Of course I’ll blog about the conference, though I can’t promise it will be at quite the level of detail I managed last year. Also, I’m glad that SIGIR is continuing to have an industry track, and I am impressed with the program that David Harper and Peter Schäuble have put together. Needless to say, I’m glad to not have the stress of being an organizer this year! Though I’ll put in an early plug for CIKM 2011 in Glasgow, where I’ll be organizing the industry track with former co-worker Tony Russell-Rose.

Some SIGIR papers that caught my attention in the program:

• Predicting Search Frustration
  Henry Feild, James Allan (University of Massachusetts Amherst), Rosie Jones (Yahoo! Labs)
  (looks like a follow-up to the first two authors’ HCIR 2009 paper on Modeling Searcher Frustration)
• Relevance and Ranking in Online Dating Systems
  Fernando Diaz, Donald Metzler, Sihem Amer-Yahia (Yahoo! Labs)
• On Statistical Analysis and Optimization of Information Retrieval Effectiveness Metrics
  Jun Wang, Jianhan Zhu (University College London)
• Is the Cranfield Paradigm Outdated? (keynote)
  Donna Harman (NIST)
• Interactive Retrieval Based on Faceted Feedback
  Lanbo Zhang, Yi Zhang (University of California at Santa Cruz)
• Do User Preferences and Evaluation measures Line Up?
  Mark Sanderson, Monica Lestari Paramita, Paul Clough, Evangelos Kanoulas (University of Sheffield)
• Human Performance and Retrieval Precision Revisited
  Mark D. Smucker, Chandra Prakash Jethani (University of Waterloo)

As for the SimInt workshop, it aims ”to explore the use of Simulation of Interactions to enable automated evaluation of Interactive Information Retrieval Systems and Applications.” I’m very excited about this attempt to bridge the gap between TREC/Cranfield and IIR/HCIR through simulation. Props to Leif Azzopardi, Kal Järvelin, Jaap Kamps, and Mark Smucker for organizing it!

If you’re planning to attend SIGIR, please give me a shout! I plan to be there for the entire conference, and you’ll probably find me at the Google booth during some of the coffee breaks.

• Gridworks, developed by David Huynh, Stefano Mazzocchi, and their colleagues at Metaweb, the company behind Freebase.
• Needlebase, developed by Justin Boyan and colleagues at ITA Software, the company powering travel search for Kayak, Orbitz, and others.

If you’re concerned with building and maintaining collections of semi-structured data, or building your own technology for this purpose, I suggest you check out these state-of-the-art tools.

In a prediction market, any event with a quantifiable (e.g., binary) outcome can be converted into an asset. At any given time, the asset value corresponds to the market prediction of the probability of the outcome. Just as in any security market, participants determine the value through their buying and selling actions. In principle, this framework allows any event with a quantifiable outcome to be predicted by a marketplace.

But, at least from my vantage point, prediction markets have not had a broad impact on decision making, despite all of the “anys” in the previous paragraph. Outside of political forecasting and sports gambling (and of course finance itself), I’m not aware of any groups outside of academia that invest significantly in the use of  prediction markets. Sure, there’s the Hollywood Stock Exchange that applies the fantasy sports concept to the movie industry and even startup Empire Avenue that aspires to generalize this idea even further into an “online influence stock exchange”. Still, I think it’s safe to say that prediction markets have had limited traction to date.

Many people do, however, believe that we can harness the wisdom of crowds. In particular, we as consumers rely on reviews and recommendations to inform our decisions about what to buy, read, etc. Because those decisions have financial implications for sellers, the world of online reviews has an adversarial element, where review systems face manipulation by those who would shill their own products or services. As a result, it is never clear how much we as consumers should trust the reviews we read to be sincere, let alone useful.

Which brings me back to prediction markets. Unlike most venues for soliciting collective opinion, prediction markets offer a strong incentive for accuracy. Betting on whether readers will like a book is quite different than simply offering a review that asserts an opinion without any risk to the person making the assertion. It is possible to manipulate a prediction market (e.g., by flooding it with high bets), but research suggests that such manipulations are short-lived and in fact expose the manipulator to significant financial risk when the price re-stabilizes.

So why don’t we use prediction markets instead of relying on reviews and recommendations? Perhaps we should, and it’s just a matter of time until entrepreneurs build successful businesses around this idea. But I suspect that much of the value of user-generated content today comes from contributors not thinking in market terms. While using prediction markets could solve the problem of shill reviews, it might also scare off the altruists.

Still, it seems to me that we should look for more opportunities to incent accuracy. Even altruistic reviewers have an interest in establishing their credibility, at least if that credibility determines the propagation of they opinions they share (perhaps I’m conflating altruism with egotism). The challenge may be to implement a marketplace that deals in the social currency of reputation than the hard currency of cash–while avoiding the sort of virtual currency that many people see as meaningless.

Can we obtain the benefits of market dynamics and still take advantage of the less rational motivations that drive some of the best online reviews today? I hope there are people who feel incented to work on this problem!

Some previous posts for further reading:

• Thoughts About Online Reputation
• Payola? There’s An App For That!
• Are Links A Distraction? (please let me know if you like my attempt to keep links outside the main flow)

Nonetheless, I feel that links supply context, and I’m a fan of keeping context nearby. Indeed, I find that clicking on a link incurs a much lower cognitive cost than flipping to the back of the book, searching for an endnote. I’ve had readers specifically thank me for including links to Wikipedia entries for technical terms. I assume those readers are fully capable of finding those Wikipedia entries themselves, but that they appreciate the convenience of the links.

Some of the commenters on Carr’s post suggest that we use technology to address this tension between preserving the reader’s focus and supplying nearby context. Specifically, we can use CSS and have a JavaScript button that toggles the link style between visible and invisible. I like the idea of handing readers control of the presentation style, though I still think it’s important to pick a sensible default. At the very least, a document should be self-contained so that a reader can choose if and when to look at the material it cites. The document should also give credit where it’s due, linking to the material it cites in a way that is visible to people and search engines. Beyond that, I think it’s really a matter of author style.

Still, I’m curious what folks here–especially long-time readers–think. Do I link so heavily that it’s distracting? Would it be easier to read my posts if the links were in a block at the end? I write for you, so please let me know how I can make this blog better. I don’t have the resources to conduct cognitive load experiments, but I’m very receptive to comments.

The other day, I received a surprise package in the mail: a copy of IBM researchers David Carmel and Elad Yom-Tov‘s newly published lecture on “Estimating the Query Difficulty for Information Retrieval“. I wasn’t even aware that this book was being written, so I’m especially appreciative of the publisher’s kindness to send me a copy.

If you liked Claudia Hauff‘s recent dissertation on “Predicting the Effectiveness of Queries and Retrieval Systems” (cf. my blog post on how “Not All Queries Are Created Equal“), then you’ll love this compact lecture that review the work on pre-retrieval and post-retrieval prediction of query performance. It covers query clarity, ranking robustness, query coherence, and much more.

I’m a big fan of the Morgan & Claypool series of Synthesis Lectures on Information Concepts, Retrieval, and Services, though I’m admittedly biased. Still, I think these books are an excellent way to get an overview of a subject, and Carmel and Yom-Tov’s book delivers wonderfully. For those not lucky enough to receive free copies in the mail, I recommend Amazon, which is selling it for less than $24.

If you like faceted search and are interested in design patterns for it, I encourage you to check out Moritz Stefaner‘s work on elastic lists. Here is his description:

Elastic lists allow to navigate large, multi-dimensional info spaces with just a few clicks, never letting you run into situations with zero results. They enhance traditional UI approaches for facet browsers by visualizing weight proportions, animated transitions, emphasis of characteristic values and sparkline visualizations.

And the good news is that elastic lists are now an open source project, available under an Apache 2.0 license. Also available for free is a book chapter on faceted search user interface design that Stefaner co-authored last year.

I was just looking at LinkedIn and found myself pleasantly surprised by a minor UI improvement in the “People You May Know” widget: as you delete people you don’t know, the widget now updates without your having to go to another page or refresh the home page. Curious, I looked to see if LinkedIn had blogged about it.

What I discovered was an even nicer surprise: LinkedIn now connects the People You May Know feature to its faceted search interface. Indeed, they blogged about it earlier this week. Props to LinkedIn for continuing to advance the state of the art in people search!

This is just so brilliant that I had to post it here. I’ve blogged in the past about alerting spam, but this guy took the idea to a new level, with great return on investment. Perhaps the news about this story will make the tactic more popular and thus less effective through dilution. Still, it’s fun to see how people exploit inefficiencies in attention markets.

Examples:

• A question like [gdp of usa] returns a chart derived from public data.
• Searching for [dora] (yes, I have a 2-year old!) suggests elmo, mickey mouse, barney, scooby doo, and bratz as “something different”.

I’m thrilled to see my colleagues’ work  getting more visibility.

The focus of his talk, naturally, was patterns. Specifically, he advocated that we take the behavior patterns of information seekers that library and information scientists have been studying for years, and use them to inform design patterns for search user interfaces.

One point he raised that deserves a deeper dive: number of media (mobile, kiosk, TV) environments push people to browse, partly because of limitations of the medium but also taking advantage of the novelty and relative lack of user habits. Unfortunately, browsing doesn’t always scale in those environments, so search is usually available as a contingency.

Interestingly, while Peter promotes rich interfaces in many of his patterns, he noted that great results ranking plus speedy response (he uses Google “classic” as his example) does allow users to rapidly reformulate their queries while staying in the flow of the information seeking experience. He returned to Google later in his talk, noting that the new interface goes beyond ranking to support a richer user interaction.

And, like me and Marti Hearst (yesterday’s keynote), Peter advocates faceted navigation (I won’t quibble on whether to call it navigation or search) as his favorite search design pattern. He uses the NCSU library as an example not only of a great implementation but also of an organization that continues to experiment with incremental design changes. He also showed faceted search examples from other domains, including Amazon and Buzzilions.

Other patterns he discusses included question answering (his example being Wolfram Alpha) and decision making (his example being Hunch). He didn’t go deep on these, but rather invited the audience to consider a broad palette of strategies for supporting information seeking. Indeed, when I asked him about question answering, he conceded that he was a skeptic and preferred a conversational (i.e., HCIR) approach akin to a librarian’s reference interview.

His closing note was about bridging the gap between physical and digital information, where he offered a potpourri of examples (from Redbox to a tweeting plant). I work in local search, so in my case he’s preaching to the converted. But I think he’s right that everything is only recently coming together–specifically, the ubiquity of digital data on the internet and of mobile devices in the physical world that can both consume and produce that data. Many of us take these developments for granted, but it’s important that we adapt our approach to search to address what is a very recent phenomenon.

Fun stuff! I didn’t get to attend the rest of the summit, but I encourage you to check out the tweet stream at #ESS10.

Her major take-aways:

• The user’s emotional response is a key aspect of the information seeking experience.
• There is a double vocabulary problem: different ways to express same concept (cf. Furnas et al.), and users stubbornly anchor on initial query terms (cf. Kahneman, Tversky, et al.)
• Recognition is easier than recall, so interfaces need to support the recognition process.
• Don’t personalize search, socialize it!

She peppered her talk with concrete examples and scholarly references. Given that her book is available online for free, I won’t try to replicate them all here! Still, I’ll single out two Noisy Community members: FXPAL researchers Jeremy Pickens and Gene Golovchinsky (for their SIGIR 2008 work on collaborative exploratory search) and user experience designer Greg Nudelman for his proposal of faceted breadcrumbs as a search user interface.

If you missed her live, you check find a video of a tech talk she gave at Google a few months ago. You can also check out the conference tweet-stream at #ESS10.

Today I celebrate six months of working at Google. I’m having a great time, and I wanted to take a moment to share a bit about my experience thus far.

First, a bit about Google’s New York office. It is a major office–in fact, Google’s second largest office and an R&D powerhouse. New York Googlers played key roles in two of Google’s recent developments: real-time search and the results page re-design. Less visibly but not less importantly, engineers at Google New York contribute to every major aspect of Google’s technology.

My own contributions have been toward improving local search. Local search represents an increasingly large and important fraction of people’s online information seeking. At first glance, it might seem to be an easier problem than general web search, since there are only tens of millions of places, compared to tens of billions of web pages. But local search poses unique challenges–from data quality to ranking to supporting exploratory search.

Another area that I’m especially excited about is the work on structured data. The Magpie team is based in New York. You may be familiar with them as the team that developed Google Squared–which powers the “something different” links for web search.

Of course, there is far more going on at Google New York than I could hope to summarize in a blog post–including lots that I can’t talk about yet. But I hope this at least gives you a taste of what it’s like to work for the world’s best search company in the world’s best city (yes, I know I’ll take some flak for at least one of those superlatives).

If you’re interested in learning more, please don’t hesitate to reach out to me. I may have drunk the kool-aid, but I promise to be candid and as open as I can about what it’s like to go through the hiring process and what awaits you at the end of it.

Speaking of Jason, check out the latest improvements he’s made to the TunkRank interface. Pretty slick! To learn more about the TunkRank measure of Twitter influence / authority, check out this post.

I wish I could take even a gram of credit for this! I’m really proud of my colleagues for rolling out this new design that encourages and facilitates exploratory search. Go HCIR!

I’ve been doing some reading myself. Specifically, I just read F. Randall (“Randy”) Farmer and Bryce Glass‘s recent book on Building Web Reputation Systems. Given that I’ve been thinking a lot about online reviews and reputation systems (e.g., this recent post), I wanted to hear what the experts had to say.

In the book, Farmer and Glass categorize the motivations for user participation as altruistic, commercial, and egocentric. Commercial motives are clearly the most problematic: a review site loses credibility if commercially motivated reviews are disguised to make their commercial motives. Most review site scandals arise from this kind of deception (e.g., this one, this one, and  this one).

Sincerity is a necessary but insufficient condition for a review to be valuable to the person who reads it. There is still the “people like me” problem: sincere reviewers may still be uninformed, unreasonably biased, or may simply not share our tastes. User-generated content is an inherently subjective medium.

Given these challenges, it’s a wonder that online review sites work at all! And yet there are real success stories. My personal favorite is Amazon.com. While it has has its hiccups, Amazon nonetheless serves as a poster child for creating value by aggregating user opinions about products.

Amazon has a well-designed review policy that gets many key elements right:

• Reviewers have identities tied to purchasing history. That encourages disclosure (people use their real names) and discourages abuse.
• The reviews themselves–and even comments in discussion threads about individual reviews–are themselves reviewed as helpful or not. That may seem overly meta, but it does a lot to mitigate information overload.
• Grounding in objective information (product content, sales rank) reduces the ability to manipulate product perception through reviews.

The system isn’t perfect, but it’s good enough to be very useful.

But products aren’t the only reputable entities, to use Farmer and Glass’s term. What about service businesses, such as restaurants, gyms, etc. Or people?

If Amazon exemplifies online product reviews, then Yelp is the canonical example of a review site for service businesses. And, despite its own share of controversy, it is quite successful. But I dare say not quite as successful as Amazon. Part of the problem is that is demographics are less representative of the general online population (here’s what Quantcast says about Yelp and Amazon demographics for their US users). Also, there’s more variance in experiencing a service than in experiencing a product.

But Yelp has also has had  a credibility problem regarding which reviews they allow to be published. Perhaps the root of this problem is that Yelp’s business model depends on paid advertising from the businesses reviewed on the site, while businesses would much rather have unpaid positive reviews. In contrast, Amazon makes its money buy selling products–which at least makes it perceived to be more evenhanded.

But neither Amazon and Yelp have touched the third rail of online reputation: people. LinkedIn dabbles in this space by allowing its members to review one another, but reviewees have veto power over reviews–making the review graph more of a mutual admiration society.

A recent startup, Unvarnished, is trying to create a review site with teeth. Farmer argues on his blog that Unvarnished is breaking some major  rules:

• It displays negative karma–that is, it allows people to write negative reviews of one another and displays those reviews.
• The reviews are not clearly tied to context (e.g., were the reviewer and reviewee co-workers?).
• The anonymity of reviewers does not incent altruistic or even egocentric behavior, and is thus a recipe for abuse.

I’m not as down on Unvarnished as Farmer, but I agree it will have an uphill battle to succeed. Ironically,  for all of the public concern about Unvarnished becoming a trollfest, the reviews skew strongly positive. This is probably an artifact of how Unvarnished is growing its membership: current users ask friends to review them.

I agree most with Farmer that Unvarnished’s incentive structure seems problematic. A person’s friends will probably be inclined to write positive reviews, and may even be annoyed at having to write them anonymously. A person’s enemies may be inclined to write negative reviews as a form of attack or revenge. But it’s less clear what will incent people to write accurate reviews–or what will signal to readers that a review is trustworthy.

All in all, I think that these are early days in the online reputation space, and that there is ample room for innovation. Facebook’s recent release of “like buttons” is an ambitious attempt to boil the ocean of “social objects”. A best poster award at the recent WWW 2010 conference went to Paul Dütting, Monika Henzinger, and Ingmar Weber’s “How much is your Personal Recommendation Worth”.  Hopefully all of these attempts to research and innovate will lead to a world where we can derive real value from others’ opinions and feel incented to contribute our own.

Anyway, if you haven’t heard about Quora yet, here’s a summary from the site:

Quora is a continually improving collection of questions and answers created, edited, and organized by everyone who uses it.

For those of you who studied Latin, the title of this post hopefully triggers at least a faint memory of relative pronouns and declensions. It’s been suggested that “quora” is a faux-Latin plural of quorum, which in turn is the genitive plural of qui. A less arcane possibility is that quora is intended to evoke the modern-day meaning of “quorum”: a gathering of the minimum number of people of an organization to conduct business. Or perhaps “quora” is a contraction of question or answer, befitting a question-and-answer site.

How did I come up with all of these possibilities? Well, I did study Latin (semper ubi sub ubi!), but I found all of the above from the Quora entry entitled “What does Quora mean?” (membership required to view). Indeed, Quora is a great place to learn about Quora, as well as about Aardvark, Hunch, and other question-and-answer startups. Because it’s been launched as a private beta and virally marketed among friends, the community–and thus its interests–are highly skewed towards tech startups. Indeed, people seem more inclined to compare it to programmer-oriented site Stack Overflow than to Yahoo! Answers–which speaks volumes about the current user base.

All that said, is Quora a useful site? It certainly offers useful information, but that’s a pretty low bar–after all, the open web already offers lots of useful information. The better question is what Quora offers that the open web does not.

Indeed, the closed nature of the site puts it at a disadvantage to the open web: no links, search engine optimization, etc. That said, I also haven’t seen spam or any of the other abuse endemic to the open web.

In any case, I don’t see Quora as a knowledge base of first resort–except possibly to learn more about software startups. Whether by design or by virtue of its early membership, the site is a very narrow scope.

The more interesting value proposition of Quora is the community it is creating. Quora facilitates conversation, much like a members-only blog where everyone uses their real names. It’s a well-designed social site, and I like that it revolves around substantive topics.

But I worry that Quora faces a catch-22. If the focus stays narrow, I can’t imagine it creating enough utility to justify its $86M valuation. But it’s not clear that Quora can scale up to a broader scope. Given what I’ve seen of question-and-answer sites, I’m skeptical.

What Quora does have going for it is an all-star team, and I’m sure they have big plans for the site. I’m very curious to see what those plans are, and how they play out.

Also, participants building their systems in Solr can take advantage of the scripts that Tommy Chheng has prepared. Of course, you are welcome to use your own system. Commercial software companies are especially encouraged to show off their HCIR wares!

I know there’s lots of interesting stuff coming out at the Chirp Twitter developer conference this week, and I’m still catching up on it all. But I am happy to point folks to a Google Labs application that was announced this morning: Follow Finder.

It’s not the first application to suggest Twitter followers based on analysis of the social graph, but I’ve actually found its suggestions to be quite plausible. For example, it suggests @fredwilson, @cshirky, @mattcutts, @peteskomoroch, and @msftresearch as “tweeps” I should follow, and suggests that the following users have similar followers to mine: @endeca, @lemire, @yahooresearch, @googleresearch, and @mattcutts.

There’s a bit of an “everything sounds like Coldplay” effect (e.g., @fredwilson shows up in a lot of the searches I tried), but overall I’m impressed with the quality, especially compared to the other suggestion tools I’ve tried.

• Program Committee chair Rob Capra has assembled a program committee that includes Susan Dumais, Gary Marchionini, and many more.

• Folks who want to participate in the HCIR Challenge should contact me for details! You’ll have to obtain your own copy of The New York Times Annotated Corpus from the Linguistic Data Consortium (thanks to them and to The New York Times for making this data set freely available!). After that, you can roll your own application, or you can bootstrap on the Solr indexer that Tommy Chheng has graciously put together.

More details–particularly about the Challenge–are forthcoming and will be posted on the HCIR Challenge page. Meanwhile, feel free to ask me questions, either publicly here or by email, and I’ll be happy to answer them. Looking forward to seeing many of you this August!

This just in: Fernanda Viégas and Martin Wattenberg, two of the biggest rock stars in the world of data visualization (their long list of accomplishments includes Many Eyes and the Baby Name Voyager), have left IBM to form their own company, Flowing Media, headquartered in Cambridge, MA. As they wrote me, “if you know of anyone who has interesting data and would like help bringing it to life, spread the word.”

I am very excited for them, and am eagerly anticipating the work they will produce as free agents.

I haven’t talked much about TunkRank in the past months, largely because Jason Adams, who stepped up to the TunkRank Implementation Challenge last year, has been leading the charge. Indeed, all I did, beyond lending my first syllable to its name, was to propose the measure and get it implemented “Tom Sawyer” style.

Since then:

• TunkRank was cited in a WSDM 2010 paper entitled “TwitterRank: finding topic-sensitive influential twitterers“
• Nicolas Cerrato implemented an influence measure for gamer site Gamocracy based on TunkRank.

And, most recently:

• University of Oviedo professor Daniel Gayo-Avello published a research paper entitled “Nepotistic Relationships in Twitter and their Impact on Rank Prestige Algorithms“, based on a follower graph of 1.8M Twitter users, in which he reports:
  
  Lastly, there are one method clearly outperforming PageRank with respect to penalization of abusive users while still inducing plausible rankings: TunkRank. It is certainly similar to PageRank but it makes a much better job when confronted with “cheating”: aggressive marketers are almost indistinguishable from common users –which is, of course, desirable; and spammers just manage to grab a much smaller amount of the global available prestige and reach lower positions –although they still manage to be better positioned than average users. In addition to that, the ranking induced by TunkRank certainly agrees with that of PageRank, specially at the very top of the list, meaning that many users achieving good positions with PageRank should also get good positions with TunkRank. Thus, TunkRank is a highly recommendable ranking method to apply to social networks: it is simple, it induces plausible rankings, and severely penalizes spammers when compared to PageRank.
  
  You can read a summary version in his blog post, descriptively titled “Research on a 1.8M Twitter user graph. Conclusion: TunkRank is your best option.“

I’ve excited that an idea I came up with on a whim (or perhaps out of excessive idealism) has taken such a life of its own. And hey, I do work for a company that is into real-time search and that knows a thing or two about adversarial information retrieval. Hopefully I’ll find way to apply TunkRank–or at least its intuition–in my own work. In the mean time, I offer those who have already done so my congratulations and gratitude.

Regular readers may recall hearing about the New York Times Annotated Corpus (which is the basis for the HCIR Challenge), and decision to publish their tags as Linked Open Data, Given that linked data applications are still a bit exotic, NYT semantic technologist and Noisy Community regular Evan Sandhaus published a tutorial and example application to help you build your own. If you’d like to get your feet wet in the semantic web (and can forgive the mixed metaphor), this is an excellent opportunity.

People are very good at learning new concepts after observing just a few examples. For instance, a child will confidently point out which animals are “dogs” after having seen only a couple of examples of dogs before in their lives. This ability to learn concepts from examples and to generalize to new items is one of the cornerstones of intelligence. By contrast, search services currently on the internet exhibit little or no learning and generalization.

Bayesian Sets is a new framework for information retrieval based on how humans learn new concepts and generalize.  In this framework a query consists of a set of items which are examples of some concept. Bayesian Sets automatically infers which other items belong to that concept and retrieves them. As an example, for the query with the two animated movies, “Lilo & Stitch” and “Up”, Bayesian Sets would return other similar animated movies, like “Toy Story“.

How does this work? Human generalization has been intensely studied in cognitive science and various models have been proposed based on some measure of similarity and feature relevance. Recently, Bayesian methods have emerged as models of both human cognition and as the basis of machine learning systems.

Bayesian Sets – a novel framework for information retrieval

Consider a universe of items, where the items could be web pages, documents, images, ads, social and professional profiles, publications, audio, articles, video, investments, patents, resumes, medical records, or any other class of items we may want to query.

An individual item is represented by a vector of features of that item.  For example, for text documents, the features could be counts of word occurrences, while for images the features could be the amounts of different color and texture elements.

Given a query consisting of a small set of items (e.g. a few images of buildings) the task is to retrieve other items (e.g. other images) that belong to the concept exemplified by the query.  To achieve the task, we need a measure, or score, of how well an available item fits in with the query items.

A concept can be characterized by using a statistical model, which defines the generative process for the features of items belonging to the concept.  Parameters control specific statistical properties of the features of items.  For example, a Gaussian distribution has parameters which control the mean and variance of each feature. Generally these parameters are not known, but a prior distribution can represent our beliefs about plausible parameter values.

The score

The score used for ranking the relevance of each item x given the set of query items Q compares the probabilities of two hypotheses. The first hypothesis is that the item x came from the same concept as the query items Q. For this hypothesis, compute the probability that the feature vectors representing all the items in Q and the item x were generated from the same model with the same, though unknown, model parameters. The alternative hypothesis is that the item x does not belong to the same concept as the query examples Q. Under this alternative hypothesis, compute the probability that the features in item x were generated from different model parameters than those that generated the query examples Q. The ratio of the probabilities of these two hypotheses is the Bayesian score at the heart of Bayesian Sets, and can be computed efficiently for any item x to see how well it “fits into” the set Q.

This approach to scoring items can be used with any probabilistic generative model for the data, making it applicable to any problem domain for which a probabilistic model of data can be defined.  In many instances, items can be represented by a vector of features, where each feature can either be present or absent in the item.  For example, in the case of documents the features may be words in some vocabulary, and a document can be represented by a binary vector x where element j of this vector represents the presence or absence of vocabulary word j in the document.  For such binary data, a multivariate Bernoulli distribution can be used to model the feature vectors of items, where the jth parameter in the distribution represents the frequency of feature j.  Using the beta distribution as the natural prior the score can be computed extremely efficiently.

Automatically learns

An important aspect of Bayesian Sets is that it automatically learns which features are relevant from queries consisting of two or more items. For example, a movie query consisting of “The Terminator” and “Titanic” suggests that the concept of interest is movies directed by James Cameron, and therefore Bayesian Sets is likely to return other movies by Cameron. We feel that the power of queries consisting of multiple example items is unexploited in most search engines. Searching using examples is natural and intuitive for many situations in which the standard text search box is too limited to express the user’s information need, or infeasible for the type of data being queried.

Uses

The Bayesian Sets method has been applied to diverse problem domains including: unlabelled image search using low-level features such as color, texture and visual bag-of-words; movie suggestions using the MovieLens and Netflix ratings data; music suggestions using last.fm play count and user tag data; finding researchers working on similar topics using a conference paper database; searching the UniProt protein database with features that include annotations, sequence and structure information; searching scientific literature for similar papers; and finding similar legal cases, New York Times articles and patents.

Apart from web and document search, Bayesian Sets can also be used for ad retrieval through content matching, building suggestion systems (“if you liked this you will also like these” which is about understanding the user’s mindset instead of the traditional “people who liked your choice also liked these”) and finding similar people based on profiles (e.g. for social networks, online dating, recruitment and security). All these applications illustrate the countless range of problems for which the patent-pending Bayesian Sets provides a powerful new approach to finding relevant information. Specific details of engineering features for particular applications can be provided in a separate post (or comments).

Interactive search box

An important aspect of our approach is that the search box accepts text queries as well as items, by dragging them in and out of the search box.  An implementation using patent data is at http://www.xyggy.com/patent.php.  Enter keywords (e.g., “earthquake sensor”) and relevant items to the keywords are displayed.  Drag an item of interest from the results into the search box and the relevance changes.  When two or more items are added into the search box, the system discovers what they have in common and returns better results.  Items can be toggled in/out of the search by clicking the +/- symbol and items can be completely removed by dragging them out of the search box.  Each change to an item in the search box automatically retrieves new relevant results.  A future version will allow for explicit relevance feedback.  Certain data sets also lend themselves to a faceted search interface and we are working on a novel implementation in this area.

In our current implementation, items are dragged into the search box from the results list, but it is easy to see how they could be dragged from anywhere on the web or intranet.  For example, a New York Times reader could drag an article or image of interest into the search box to find other items of relevance. There is a natural affinity between an interactive search box as described and the new generation of touch devices.

Summary

Bayesian Sets demonstrates that intelligent information retrieval is possible, using a Bayesian statistical model of human learning and generalization.  This approach, based on sets of items encapsulates several novel principles.  First, retrieving items based on a query can be seen as a cognitive learning problem; where we have used our understanding of human generalization to design the probabilistic framework.  Second, retrieving items from large corpora requires fast algorithms and the exact computations for the Bayesian scoring function are extremely fast.  Finally, the example-based paradigm for finding coherent sets of items is a powerful new alternative and complement to traditional query-based search.

Finding relevant information from vast repositories of data has become ubiquitous in modern life.  We believe that our approach, based on cognitive principles and sound Bayesian statistics, will find many uses in business, science and society.

Earlier this week, I read about Unvarnished on TechCrunch and was extremely curious about this “Yelp for LinkedIn” making a bold play in the online reputation space. My curiosity should be no surprise to folks who have read my recent posts about distributed trust networks and solicited reviews. Anyway, I decided to go straight to the source and persuaded Unvarnished CEO Peter Kazanjy to invite me to the beta.

My impression so far: they’ve done a great job of collecting profiles, but the reviews themselves are pretty sparse. Moreover, most of the reviews I’ve seen so far are positive–hardly the bloodbath that the blogosphere has been predicting. Membership is relatively non-anonymous (you need to sign in through Facebook Connect), but your actual reviews are posted anonymously.

Since Unvarnished is trying to collect reviews, it’s not surprising that the way to join the beta…is to review someone. If you want to try it out and don’t mind reviewing me (anonymously) as the price of entry, let me know, and I’ll send you an invite (we’ll have to connect on Facebook first).

p.s. No, this is not an April Fool’s Joke. At least not on my part!

Web Site

• http://www.hcir2010.org/

Workshop Chairs

• Bill Kules, The Catholic University of America
• Daniel Tunkelang, Google
• Ryen White, Microsoft Research

Program Chair

• Rob Capra, University of North Carolina at Chapel Hill

Local Arrangements Chair

• Catherine Smith, Rutgers University

Sponsors

• Endeca
• Google
• Microsoft Research

Background

HCIR combines research from the fields of human-computer interaction (HCI) and information retrieval (IR), placing an emphasis on human involvement in search activities. The HCIR workshop has run annually since 2007. The workshop unites academic researchers and industrial practitioners working at the intersection of HCI and IR to develop more sophisticated models, tools, and evaluation metrics to support activities such as interactive information retrieval and exploratory search. It provides an opportunity for attendees to informally share ideas via posters, small group discussions and selected short talks.

New for 2010: the HCIR Challenge!

New this year, we will be running the HCIR Challenge! The aim of the challenge is to encourage HCIR researchers and practitioners to build and demonstrate effective information access systems. Challenge participants will have no-cost access to a large collection of almost two million newspaper articles with rich metadata generously provided for use in this challenge by The New York Times. The focus of participation is building systems (or using existing ones) to help people search the collection interactively. Entries will be judged by an expert panel based on HCIR criteria (specifically: effectiveness, efficiency, control, transparency, guidance, fun) and also judged by workshop attendees at the event. More information on the challenge will be made available on the workshop website.

Format

We invite 4-page papers that will be reviewed by an international program committee. Papers fall into two categories: position papers describing an idea, an opinion, or early-stage research, and research papers describing a conducted research study, an implemented system, or a review of prior research. Papers will be judged based on relevance to HCIR. Idea diversity across all submissions may also be considered. The revised versions will be published on the workshop website. The workshop time will be used for what participants have told us that they found most valuable in previous events: posters and directed group discussions.

We will select 4-6 papers for presentation in a workshop panel. All other attendees are strongly encouraged to present posters during the morning “poster boaster” session. Selected HCIR challenge papers will also have an opportunity to present their work orally at the event.

Our target is to have 50-75 participants.

Possible topics for discussion and presentation at the workshop include, but are not limited to:

• Novel interaction techniques for information retrieval.
• Modeling and evaluation of interactive information retrieval.
• Exploratory search and information discovery.
• Information visualization and visual analytics.
• Applications of HCI techniques to information retrieval needs in specific domains.
• Ethnography and user studies relevant to information retrieval and access.
• Scale and efficiency considerations for interactive information retrieval systems.
• Relevance feedback and active learning approaches for information retrieval.

Demonstrations of systems and prototypes are particularly welcome.

Important Dates

• Mon June 14, 2010: Submission deadline for position papers (midnight Pacific Time)
• Fri July 16, 2010: Decisions sent to authors
• Fri July 23, 2010: Deadline for accepted participants to register
• Fri July 30, 2010: Submission deadline for camera-ready copies

Workshop Fees and Travel Support

After careful consideration, we have decided to implement a $75 workshop fee. This will help offset the workshop costs and allow us to defray expenses for 1-2 graduate students.

We also appreciate the support of our corporate sponsors. It is our hope that the continuing success of this workshop will attract additional funding in future years. If your company or organization is interested in sponsoring travel scholarships, please let us know as soon as possible.

I’ll write more about Quora when I’ve had a chance to play with it myself.

Earlier this week, I had the good fortune to attend the New York Semantic Web Meetup, which featured three excellent presentations. I’ll confess that I primarily attended the event in order to learn more about open data platform Factual, particularly to see how it compares to Freebase and Google Squared.

But I found the other two presentations, “The Evolution of Social Search“ by Nitya Narasimhan and “User Interfaces for the Semantic Web” by Duane Degler, even more compelling. Hopefully I’m not just biased because they both mentioned me in their slides!

I really wish Nitya’s session had been recorded as a video, but the slides embedded above will have to suffice for those who couldn’t see it live. Hopefully they communicate Nitya’s framing of the social search space. She does a great job of weaving together the various strands of social search: people as sensors promoting “real-time” content, social filters to reflect personalized notions of trust, and routers to leverage the collective intelligence of crowds. She gives tons of examples and links for further reading. Hopefully I’ll get her to give this talk again–with less stringent time constraints–and record it for online viewing.

I’ve been following Hunch for a while, and my impression has evolved from the initial skepticism with which I greeted it a year ago (to the day!). Given the track records of co-founders Caterina Fake and Chris Dixon, perhaps I should have expected their success at obtaining traffic and funding.  But what interests me more is that they are doing interesting things with data mining and putting a new twist on social media analytics.

For those unfamiliar with Hunch, it is a decision engine (cf. [real decision engine] vs. [decision engine]). For example, it can help you decide whether to buy an iPad or how to name your baby. While it’s not clear to me how much people are using Hunch for utility vs. entertainment, Hunch is certainly accumulating users–as well as the data that those users volunteer.

Hunch recently released two applications that mash up that data with the Twitter follower graph. The first is a “Twitter Predictor Game” that attempts to calculate your taste profile from your Twitter id and then predict how you’ll answer Hunch’s taste questions. Just to keep the game honest, you can look at the Hunch’s guess either before or after you provide your answer. The second is called “Twitter Follower Stats“: given a Twitter user, it reports the salient information it has inferred about that user’s followers (e.g., @maddow vs. @karlrove).

I think this stuff is neat, and a great testament to the “unreasonable effectiveness of data“. The question-answer data still feels a bit sparse for my taste, and I suspect there’s still room for more dimensionality reduction. I’m sure Hunch CTO Matt Gattis and colleagues are working on it! Also, it would be neat to direct the follower analytics rather than simply see the ones that Hunch deems most salient.

In summary, Hunch is keeping it interesting. Definitely a startup to watch and learn from.

Mathew Ingram posted an interview with Craig Newmark (the Craig of craigslist fame) in which the latter argued that what the web needs is a “distributed trust network” to manage our online reputations. As it happens, this is an idea that has occupied me for several years. So I figured it was about time that I shared my thoughts on the subject.

When we think of how trust works online, two of the most prominent examples are Google’s PageRank measure and eBay’s feedback scores. But neither of these measures addresses what I think Craig has in mind. PageRank is a great way of using citation analysis to determine the most authoritative citations, but the trust in a page should consider its out-links (i.e., can we trust the page not to point us to untrustworthy ones?) and not just its in-links. eBay’s feedback scores have a different problem: they count positive and negative ratings without considering the social network of buyers and sellers–and approach that is vulnerable to fraud through shill ratings. Incidentally,LinkedIn recommendations have a similar weakness if viewed in strictly quantitative terms, but the potential for abuse is mitigated by the endorsements being signed–and by their being more than just binary or numerical ratings. Incidentally, here’s a site you can use if you’re too lazy to actually write the recommendations yourself.

But I digress. Propagation of trust does seem like the perfect application to build on top of social networks. Consider any problem that involves getting advice to inform a decision. If we regularly solicit advice from our first-degree connections, then we should be able to learn over time whose advice we can trust. We can then vouch for these connections, which offers the connections who trust us a basis for trusting their second-degree connections through us. And so forth through our social network. Of course, trust is not irrevocable: loss of trust should propagate similarly.

I’ve talked about this problem with two of the leading experts on social networks, Jon Kleinberg and Prabhakar Raghavan, and as far as I know no one has built a system along these principles. In economic terms, I envision a system where a person’s reputation truly is his or her coin. One person might think of bribing one another to exploit the latter’s established reputation, but a rational person with a strong reputation would demand an exorbitant bribe to put that reputation at risk.

Of course, a lot of information would have to propagate throughout the social network–and be stored–for this system to work. Regardless of how the information is abstracted, such a reputation index would raise thorny privacy issues. Nonetheless, I don’t know if we can build a reputation system that is entirely privacy-preserving–since reputation is an inherently public mechanism. In addition, any such system would have to consider the implications of defamation laws. These are some major hurdles!

Nonetheless, I agree wholeheartedly with Craig that a distributed trust network could be “the killingest of killer apps”. I just hope we can find a way to build and use it!

Note: Chris Rines suggested I look at Advogato’s Trust Metric, and a quick investigation led me to the Wikipedia entry for trust metric. Looks like I have some homework to do!

In a post on ReadWriteWeb, Sarah Perez summarizes “Measuring User Influence in Twitter: The Million Follower Fallacy“, a recent research paper by Meeyoung Cha, Hamed Haddadi, Fabricio Benevenuto, and Krishna Gummadi. The punch line should hardly be surprising to regular readers here given my variety of rants on the subject: follower count isn’t great measure of influence.

The authors focus on measuring three quantities: followers (which they call indegree), retweets, and mentions. Their main results is that, while the number of followers is strongly correlated to the numbers of retweets and mentions for the general user population, the correlation is much weaker for the users with high follower counts, e.g., in the top 10%. Indeed, the authors believe that the correlation for the general population is “an artifact of the tied ranks among the least influential users, e.g., many of the least connected users also received zero retweet and mention.”

The authors further note that:

Across all three measures, the top influentials were generally recognizable public figures and websites. Interestingly, we saw marginal overlap in these three top lists. These top-20 lists only had 2 users in common: Ashton Kutcher and Puff Daddy. The top-100 lists also showed marginal overlap, as shown in Figure 1, indicating that the three measures capture different types of influence.

The authors ultimately conclude that:

• Follower count represents a user’s popularity, but is not related to notions of influence such as engaging audience, i.e., retweets and mentions.
• Retweets are driven by the content value of a tweet, favoring mainstream news organizations.
• Mentions are driven by the name value of the user, favoring celebrities.

I can’t argue with any of the above, but I do wonder if any of them are ideal measures of influence. All three measures are easy to game–and none of them model the scarcity of user attention, which is the motivating principle of TunkRank. Nor do they ground “influence” in any outcome external to Twitter.

Still, it’s an interesting negative result. If nothing else, it helps reinforce the argument that follower count isn’t a useful measure–at least once you get beyond the very low end of the range.

It used to be a surprise when people remembered our birthdays, but in the twenty-first century Facebook ensures that we will never forget a birthday. Does that make the happy birthday wishes any less sincere? Or is technology simply providing us with a cognitive assist and helping us express our sincere feelings?

A related question is how we should react to solicited reviews–a topic that was the subject of a recent interview on Mike Blumenthal’s blog. To be clear: I’m not talking about businesses offering incentives to reviewers–most folks seem to agree that incented reviews are a bad idea. And let’s not get started on hiring interns or Turkers to write them! Rather, the question is whether a review is less meaningful because it was solicited by a business  rather than spontaneously volunteered  by the reviewer.

I’m ambivalent. I don’t think the content of a solicited review is inherently insincere–after all, the reviewers have no reason to lie. In fact, soliciting a review from a disgruntled customer may annoy that customer enough to elicit one that is scathingly sincere!

Nonetheless, it’s hard to imagine why a business wouldn’t target a solicitation campaign at a sympathetic set of reviewers, given the opportunity to do so. Given that consumers put a fair amount of trust in aggregated reviews (as documented in the Forrester study about the “groundswell effect“), skewing the population of reviewers can significantly stack the desk.

And even a uniform campaign to solicit reviews raises concerns. Research by Yong Liu supports the adage that all buzz is good buzz–though in fairness I don’t know if he observed causality or just correlation. But I can extrapolate from personal experience that the number of reviews signals the popularity of a product or service. And I doubt I’m alone, given that Yelp, MenuPages, and other review sites let you filter or sort by the number of reviews. A successful campaign to solicit reviews, even if it doesn’t skew the polarity of the reviews, will at least inflate their quantity.

Still, where’s the harm? There’s nothing unethical in a business soliciting private or public feedback. And, back to the birthday example, I haven’t seen anyone upset by Facebook-prodded birthday greetings. Perhaps the online solicitation of reviews serves a similar “reminder” purpose, and we should simply accept its as part of our twenty-first century reality.

But consumers will need to re-calibrate their trust in reviews–or at least in what the numbers signal–if it turns out that a significant fraction of them are solicited. As Yelp CEO Jeremy Stoppelman pointed out in a blog post defending his company against recent legal action:

If a business could garner a top rating on Yelp simply by soliciting 5-star reviews from friends, family, and favored customers, how useful would such a site be?

While I don’t know enough to comment on the legal merits of the lawsuits (or the history of allegations that Yelp extorts advertising from businesses), I can understand how a proprietary review filter is controversial and invites skepticism from businesses whose positive reviews are filtered or demoted–especially given that relevance ranking raises similar concerns. But I can also understand how making such a filter completely transparent could defeat its stated purpose: ”to protect consumers and business owners from fake, shill or malicious reviews”. And Yelp does at least disclose that it considers users’ activity level as a signal in its filter.

But let’s face it: it’s hard to draw a clear distinction between a solicited responses and spontaneous ones. Review sites have never claimed to conduct scientific polls, and consumers should be sophisticated enough to expect some degree of sample bias. Moreover, the process does not have to be perfect in order to be useful to consumers–we learn to approach review sites with a calibrated level of cynicism.

Still, my hope is that consumers will start placing less stock in the aggregated opinions of anonymous strangers and shift their trust to people who are more transparent about their identities and motives. The more that reviewers stand behind their opinion and put their own integrity on the line, the less it will matter whether those opinions are solicited or spontaneously expressed. We’ll see how the opinion marketplace sorts this out.

Last week, Google head of webspam Matt Cutts posted a full-throated defense of Google’s transparency on Google’s European Policy Blog in response to complaints that a few companies raised to the European Commission. Long-time readers of my blog know that I’m a big fan of search engine transparency and have made my own calls on this blog for Google to be more transparent. The fact that I work at Google now doesn’t change my values. But being on the inside has informed my perspective.

In particular, as Matt elaborates in his post, Google deserves more credit for transparency than it often gets from its critics. For example, Google has published:

• “The Anatomy of a Large-Scale Hypertextual Web Search Engine“, which not only details the formula for PageRank, but also mentions other signals that Google uses to rank search results: anchor text, location of query terms within documents, proximity of query terms, etc.
• details of its key infrastructure innovations: MapReduce, the Google File System, Bigtable, and Protocol Buffers
• hundreds of research papers by Googlers in diverse areas of computer science

He goes onto describe the various webmaster tools and social media resources that Google has made available. The popularity of these tools is a testament to their utility.

Still, as Matt points out:

we don’t think it’s unreasonable for any business to have some trade secrets, not least because we don’t want to help spammers and crackers game our system. If people who are trying to game search rankings knew every single detail about how we rank sites, it would be easier for them to ‘spam’ our results with pages that are not relevant and are frustrating to users — including porn and malware sites.

As I blogged back in 2008, I still hope that someday we won’t need to have to rely on a relevance analog of security through obscurity in order to deter spam and abusive SEO practices. But I recognize that we haven’t developed such an analog, and hence that complete transparency today for web search ranking algorithms would have a far greater downside than upside for ordinary users.

I suspect that a prerequisite for complete transparency in search requires moving from a ranking-based retrieval approach to a set-based approach. For many web search information needs (e.g., navigational queries), it’s hard to see how users would benefit from such a radical change. For queries that represent more exploratory information needs, a set-based approach would be (at least in my view) far preferable to one based on ranking. But there’s a lot of work to do on the content side before such exploratory interfaces for the web are usable.

In summary, I’m happy to see Matt taking a public stand in Google’s defense. I don’t always agree with my employer’s decisions, but I do believe that my colleagues act in good faith and with good intentions. I understand how many people–especially site owners–fixate on whatever Google keeps secret. In a world where so many people compete for attention, information is power. Google tries to provide maximum quality to users while keeping the playing field level for site owners. As Google Fellow Amit Singhal points out, “this stuff is tough“.

I was particularly interested in work related to the query clarity score initially proposed by Steve Cronen-Townsend, Yun Zhou, and Bruce Croft in a 2002 paper entitled “Predicting Query Performance“. But there is a wide variety of work in this area, including methods to predict performance either before or after results retrieval.

Happily, Claudia Hauff just published a dissertation on this topic, entitled “Predicting the Effectiveness of Queries and Retrieval Systems“. It is very well written, and I recommend it to anyone interested in learning more about this subject. She presents not only her own original research, but also a comprehensive analysis of others’ efforts.

Here is an excerpt from the abstract:

In this thesis we consider users’ attempts to express their information needs through queries, or search requests and try to predict whether those requests will be of high or low quality. Intuitively, a query’s quality is determined by the outcome of the query, that is, whether the retrieved search results meet the user’s expectations. The second type of prediction methods under investigation are those which attempt to predict the quality of search systems themselves. Given a number of search systems to consider, these methods estimate how well or how poorly the systems will perform in comparison to each other.

I look forward to seeing researchers continue to build on these results, and I am excited for the day when search engines are more reflective on their own strengths and weakness.

The Fourth Workshop on Human-Computer Interaction and Information Retrieval will take place on August 22nd at Rutgers University in New Brunswick, NJ. It will be an independent workshop as in previous years, but this year we are co-locating it with the Third Information Interaction in Context conference (IIiX 2010).

We’ve already lined up Google’s Dan Russell as a keynote speaker and are close to circulating a call for participation. We’re also planning to introduce something new to the workshop this year: an HCIR challenge! Participants will build applications around a specific data set that demonstrate the use of HCIR techniques. We’ll announce the data set and the challenge details as soon as we’ve confirmed the licensing details.

Meanwhile, we’re looking for a volunteer to help us build a baseline index for the challenge data set. Participants will be allowed–but not required–to use this index as a starting point for their entries. The volunteer should be comfortable using open-source packages Lucene or Solr. If you are interested in being that volunteer, please let me know, and I’ll be happy to share more details.

in an information-rich world, the wealth of information means a dearth of something else: a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it (Simon, 1971)

I hope everyone agrees that attention is a scarce good. But I’m curious how people measure it. After all, if we’re going to talk about an economic good being scarce, we ought to quantify it!

One approach is to measure attention at a specific moment in time, measuring how much of our instantaneous cognitive capacity we devote to a task. This approach is useful for evaluating a user interface–in particular, for determining how users allocate their attention among the various interface elements. Another approach is to measure attention in units of time, e.g., how many of our waking hours do we devote to a particular activity. This latter strikes me as more of what Herb Simon had in mind.

We can interpret the two definitions as equivalent–after all, cumulative attention devoted to a task is simply the sum (or integral) of instantaneous attention over time. But thinking this way so misses a key consideration: we pay a significant price for context switching.

A familiar example is email. The total time we spend reading email is a productivity concern, but the larger concern for many of us is the frequency with which email causes us to interrupt our workflow. Knowing this, I made a brief attempt in 2008 to check email only once a day. Unfortunately, this approach would have violated too many of my peers’ expectations. I returned to status quo, reading my email (or at least scanning headers) as it arrives. Other messaging tools, such as instant messaging and Twitter, only add to the challenge of managing our personal communication flow.

Of course, what I really want is for my messaging tools to distinguish urgent messages from non-urgent ones, and to only interrupt my workflow for the former. I know that no system, whether based on manual filtering or algorithmic analysis, can make this subjective classification with 100% accuracy, but I’d certainly accept a handful of false positives in exchange for far fewer interruptions. I suspect I’m not alone.

Moreover, this approach extends beyond personal communications to more public ones, such as social media platforms and even web search. On one hand, the passing of time offers an opportunity to accumulate reliable content analysis; on the other hand, we don’t want to miss time-sensitive content just because the system waited too long to determine the content’s relevance to our information needs. Still, the low signal-to-noise ratio on social media platforms suggests to me that many information consumers would be amenable to a different tradeoff than the one we experience today.

What I’d really like to see is systems take advantage of the differences in users’ personal senses of urgency. Some examples:

• A widely broadcast email isn’t delivered all at once, but first goes to users with higher urgency settings. Because those users mark it as spam, the email is already marked as spam for users with lower urgency settings. Conversely, if enough high-urgency users mark it as important, then it may be sent to lower-urgency users sooner.
• High-urgency users frequently check news sites and blogs. If an article attract a threshold level of engagement from high-urgency users, then low-urgency users are notified. This approach could apply to general news or to news in a specific topic that the user follows.
• Same as above, but applied to activity feeds and based on engagement within your social network. But again, high-urgency users lead the way, seeing updates sooner but at the price of experiencing a noisier stream.

To some extent, our existing systems already approximate this approach. Mechanisms like favoriting and re-tweeting propagate signal from information scouts to their followers, as do algorithms that rank real-time information based on engagement. Still, as an information consumer, I’d appreciate an interface that explicitly and transparently adapts to my priorities, and that manages interruption of my workflow accordingly.

What do folks here think? Is information delayed tantamount to information denied? Or is time on our side, potentially offering us a better tradeoff than the one we experience today?

Amazon Mechanical Turk is a marketplace for work that requires human intelligence. The Mechanical Turk web service enables companies to programmatically access this marketplace and a diverse, on-demand workforce. Developers can leverage this service to build human intelligence directly into their applications.

But, in my view, Mechanical Turk does not take its vision far enough. In the conditions of use, Amazon makes it clear that only human participation need apply: “you will not use robots, scripts or other automated methods to complete the Services”.

On one hand, I can understand that Amazon’s vision for Mechanical Turk, like Luis Von Ahn‘s “games with a purpose“, explicitly aims to apply human intelligence to tasks where automated methods seem inadequate. On the other hand, what are automated methods but encapsulations of human methods? It seems odd for Amazon to be so particular about the human / machine distinction, especially given that terms of service impose practically no other constraints on execution (beyond the obvious legal ones), Moreover, Mechanical Turk offers developers a variety of ways to assure quality (redundancy, qualification tests, etc.).

Granted, there are some important concerns that would have to be addressed if Amazon were to relax the “humans-only” constraint. For example, a developer today can reasonably assume that two different human “Providers” execute tasks independently. With automated participation, there’s a far greater risk of dependence–e.g., from multiple programmers applying the same algorithms. This possibility would have to be taken into account in quality assurance.

Still, the benefits of allowing automated participants would seem to far outweigh the risks. At pennies a task, Mechanical Turk has a limited appeal to the human labor force–indeed, research by Panos Ipeirotis suggests that Amazon’s revenue from the service may be so law that it doesn’t even cover the costs of a single dedicated developer!

In contrast, there’s evidence that programmers would take an interest in participation, were it an option. Marketplaces like TopCoder and competitions like the Netflix Prize suggest that computer scientists take an interest in proving their mettle in many of the kinds of tasks for which organizations already use Mechanical Turk.

So, why not give algorithms a chance? Surely we’re not that afraid of Skynet or the “technological singularity“. Let’s give machines–and their programmers–a chance to show off the best of both worlds!

While I catch up, I thought I’d share with you a demo that Eric Iverson was gracious enough to share with me. It uses Yahoo! BOSS to support an exploratory search experience on top of a general web search engine.

When you perform a query, the application retrieves a set of related term candidates using Yahoo’s key terms API. It then scores each term by dividing its occurrence count within the result set by its global occurrence count–a relevance measure similar to one my former colleagues and I used at Endeca in enterprise contexts.

You can try out the demo yourself at http://www.ittybittysearch.com/. While it has rough edges, it produces nice results–especially considering the simplicity of the approach.

Here’s an example of how I used the application to explore and learn something new. I started with ["information retrieval"]. I noticed “interactive information retrieval” as a top term, so I used it to refine. Most of the refinement suggestions looked familiar to me–but an unfamiliar name caught my attention: “Anton Leuski”. Following my curiosity, I refined again. Looking at the results, I immediately saw that Leuski had done work on evaluating document clustering for interactive information retrieval. Further exploration made it clear this is someone whose work I should get to know–check out his home page!

I can’t promise that you’ll have as productive an experience as I did, but I encourage you to try Eric’s demo. It’s simple examples like these that remind me of the value of pursuing HCIR for the open web.

Speaking of which, HCIR 2010 is in the works. We’ll flesh out the details over the next weeks, and of course I’ll share them here.

Just letting readers know that I’ll be on vacation for the next week. If you are starved for reading materials, check out some of the blogs I read.

Today is the last day of WSDM 2010, and I unfortunately spent it at home drinking chicken soup. But I’ve been following the conference via the proceedings and tweets.

The day started with a short session on temporal interaction. Topics included clustering social media documents (e.g., Flickr photos) based on their association with events, statistical tests for early identification of popular social media content, and analysis of answers sites (like Yahoo! Answers) as evolving two-sided economic markets.

The next session focused on advertising. Two papers focused on click prediction: one proposing an Bayesian inference model to better predict click-throughs in the tail of the ad distribution; the other presenting a framework for personalized click models. Another paper addressed the closely related problem of predicting ad relevance. The remaining papers discussed other aspects of search advertising: one on estimating the value per click for channels like Google AdSense, where ad inventory is supplied by a third party; the other proposing an algorithmic approach to automate online ad campaigns based onlanding page content.

The following session was on systems and efficiency, a popular topic given the immense data and traffic associated with web search. Two papers proposed approaches to help short-circuit ranking computations: one by optimizing the organizations of inverted index entries to consider both the static ranks of documents and the upper bounds of term scores for all terms contained in each document; the other using early-exit strategies to optimize ensemble-based machine learning algorithms. Another used machine learning to mine rules for de-duplicating web pages based on URL string patterns. Another focused on compression, showing that web content is at least an order of magnitude more compressible that what can be achieved by gzip. The last paper proposed a method to perform efficient distance queries on graph (i.e., web graphs or social graphs) by pre-computing a collection of node-centered subgraphs.

The last session of the conference discussed various topics in web mining. One presented a system for identifying distributed search bot attacks. Another proposed an image search method using a combination of entity information and visual similarity. The final paper showed that shallow text features can be used for low-cost detection of boilerplate text in web documents.

All in all, WSDM 2010 was an excellent conference, and I’m sad to not to have been able to attend more of it in person. I’m delighted to see an even mix of academic and industry representatives sharing ideas and working to make the web a better place for information access.

Unfortunately, I woke up this morning rather under the weather, so I’m having to resort to remotely reporting on the second day of WSDM 2010 conference, based on the published proceedings and the tweet stream.

The day started with a keynote from Harvard economist Susan Athey. Her research focuses on the design of auction-based markets, a topic core to the business of search which largely relies on auction-based advertising models (cf. Google AdWords). Then came a session focused on learning and optimization. One paper proposed a method to learn ranking functions and query categorization simultaneously, reflecting that different categories of queries leads users to have different expectations about ranking. Another combined traditional list-based ranking with pair-wise comparisons between results to separate the results into tiers reflecting grades of relevance. An intriguing approach to query recommendation treated it as an optimization problem, perturbing users’ query-reformulation path to maximize the expected value of a utility function over the search session. Another paper looked not at ranking per se, but rather at improving the quality of training data for using machine learning for ranking. The final paper of the session, which earned a best-paper nomination, modeled document relevance based not on click-through behavior, but rather on post-click user behavior.

The next session was about users and measurement. It opened with another best-paper nominee: a analysis of over a hundred million users to understand how they re-find web content. Another offered a rigorous analysis of the often sloppily presented “long-tail” hypothesis: it found that light users disproportionately prefer content at the head of distribution while heavy users disproportionately prefer the tail. Another log-analysis paper analyzed search logs using a partially observable Markov model, a variant of thehidden Markov model in which not all of the hidden state transitions emit observable events–and compared the latent variables with eye-tracking studies. An intriguing study demonstrated that user behavior models are more predictive of goal success than models based on document relevance. The final paper of the session proposed methods for quantifying the reusability of the test collections that lie at the heart of information retrieval evaluation.

The last session of the day focused on social aspects of search. Two of the papers were concerned with modeling authority and influence in social networks, a problem in which I take a deep personal interest. Another inferred attributes of social network users based on those of other users in their communities (cg. MIT’s Project Gaydar). Another analyzed Flickr and Last.fm user logs to show that users’ semantic similarity based on their tagging behavior is predictive of social links. The final paper tackled the sparsity of social media tags by inferring latent topics from shared tags and spatial information.

Not surprisingly, a disproportionate number of contributors to the conference work at major web search companies, who have both the motivation to improve results and the access to data that is needed for such research. One of the ongoing research challenges for the field is to find ways to make this data available to others while respecting the business concerns of search engine companies and the privacy concerns of their users.

Today was the first day of the Third ACM International Conference on Web Search and Data Mining (WSDM 2010), held at the Polytechnic Institute of NYU in Brooklyn, NY. WSDM is a young conference that has already become a top-tier publication venue for research in these areas. In contrast to some of the larger conferences, WSDM is single-track and feels more intimate and coherent–even with over 200 attendees.

The day started with an ambitious keynote by Soumen Chakrabarti (IIT Bombay): “Bridging the Structured Un-Structured Gap”. He described a soup-to-nuts architecture to annotate web documents and perform complex reasoning on them using a structured query language. But perhaps this ambitious approach is a practical one: it uses the web we have–as opposed to waiting for the semantic web to emerge–and there is a prototype using half a billion documents.

The first paper session focused on web search. Of the five papers, two emphasized temporal aspects of content, one considered social media recommendation, and one focused on identifying concepts in multi-word queries. The last paper of the session proposed using anchor text as a more widely available input than query logs to support the query reformulation process. It also attracted the most audience attention–whileinteraction is often a niche at information retrieval conferences, it always elicits strong interest and opinions.

The following session focused on tags and recommendations. Some take-aways: users produce tags similar to the topics designed by experts; individual “personomies” can be translated into aggregated folksonomies; matrix factorization methods can produce interpretable recommendations.

The last session of the day covered information extraction. One of the papers used pattern-based information extraction approaches, demonstrating how far we’ve come since Marti Hearst‘s seminal work on the subject. Another offered a SQL-like system for typed-entity search, complete with a live, publicly accessible prototype. The final paper addressed an issue the came up repeatedly at the SSM workshop: the problem of distilling the truth from a collection of inconsistent sources.

It is my pleasure to report on the 3rd Annual Workshop on Search in Social Media (SSM 2010), a gathering of information retrieval and social media researchers and practitioners in an area that has captured the interest of computer scientists, social scientists, and even the broader public. The one-day workshop took place at the Polytechnic Institute of NYU in Brooklyn, NY, co-located with the ACM Conference on Web Search and Data Mining (WSDM 2010). The quality of the presenters, the overbooked registration, and the hundreds of live tweets with the #ssm2010 hashtag all attest to the success of this event.

The workshop opened with a warm welcome from Ian Soboroff (NIST), immediately followed by a keynote from Jan Pedersen, Chief Scientist of Bing Search. Jan established a clear business case for search in social media: the opportunity to deliver content that is fresh, local, and under-served by general web search. He drilled into particular types of content where social media search is most useful: expert opinions, breaking news, and tail content. The benefits of social media search include trust and personal interaction (as compared to web content that is often soulless and of uncertain provenance), low latency (though perhaps at the cost of accuracy), and access to niche or ephemeral information that web search rarely surfaces. But delivering social media results to searchers creates its own variety of challenges, such as weighing freshness against accuracy and relevance, coping with loss of social content’s conversational context, managing low update latency when search engines have not been optimized for it, and fighting new kinds of spam. Despite these challenges, it is clear that the major web search engines have embraced the brave new world of real-time social content.

Eugene Agitchein (Emory University) then moderated a panel representing the world’s leading search engines: Jeremy Hylton (Google), Matthew Hurst (Microsoft), Sihem Amer-Yahia (Yahoo!), and William Chang (Baidu). Jeremy justified the universal interface approach, pointing out that users don’t want to have to figure out what kind of search site to use for their queries, and that they expect a familiar interface. He also noted that Google has made great strides on update latency: it can index the Twitter firehose in the same amount of time as serving a query. Matthew offered various analyses of the social search problem, based on whether the information signal resides in content (e.g., web) or attention (e.g., Twitter), or whether the information need is expressed in an explicit search query or inferred from the user’s context. Sihem offered a counter-point to Jeremy, arguing that social media search queries often represent broad or vague information needs, and thus call for a more browsing-oriented interface than web search, which is optimized for highly specific needs. William noted that the biggest competitive threat he sees for web search engines comes from social media players–and he credits much of Baidu’s success to its surfacing of social media content.

Then came a flurry of questions, perhaps the most interesting of which was how to address identity management. William argued that people prefer interacting with real-named (or pseudonymous) people to whom they are directly connected. Sihem offered the counter-example of obtaining recommendations through community aggregation. Matthew noted the incongruity of there being no economic relationship between social network companies that maintain proprietary social graphs and people whose identities and relationships those graph represent. Jeremy pointed out that users benefit if the data is as open as possible.

Given the almost even split between academic and industry participation in the workshop, the panelists were also asked to present research challenges to academia. Jeremy posed the problem of determining when social media results are actually true. Matthew wants to see more interdisciplinary work between computer scientists and social scientists. Sihem offered two challenge problems:  scalable community discovery and evaluation of collaborative recommendation systems. William wants to see a rigorous axiomatization of social media search behavior.

After lunch, Jeremy Pickens (FXPAL) moderated a panel representing social media / networking companies: Hilary Mason (bit.ly), Igor Perisic (LinkedIn), and David Hendi (MySpace). Hilary noted that, while bit.ly does not have access to an explicit social graph, it captures implicit connections from user behavior that may not be represented in the graph. Jeremy asked the panelists how much a person’s extended network matters; David and Igor pointed out research indicating correlations of mood and even medical conditions between people and their third-degree connections. Again, the audience was full of questions, especially for Igor. As a fan of faceted search, I was glad to see him touting LinkedIn’s success in making faceted search the primary means of performing people search on the site. For an in-depth view, I recommend “LinkedIn Search: A Look Beneath the Hood“.

The afternoon continued with a poster / demo session emphasizing work in progress: tools, interfaces, research studies, and position papers. I particularly enjoyed listening to the stream of interaction between academic researchers and industry practitioners.

The final panel session assembled academic researchers to discuss their views of the challenges in social media. Gene Golovchinsky (FXPAL) moderated a panel comprised of Meena Nagarajan (Wright State University), Liangjie Hong (Lehigh University),Richard McCreadie (University of Glasgow), Jonathan Elsas (CMU), and Mor Naaman (Rutgers University). Meena highlighted the need to build up meta-data to describe the context around social utteracnces. Liangjie took a position similar to William Cheng’s, calling for a framework to model the tasks and behavior of users who interact with social media. Richard focused on the intersection of social media and news search, and noted that some of the most useful information is private and proprietary (e.g., search and chat logs). Jonathan offered a variety of challenges: determining the right retrieval granularity, managing multiple axes of organization, aggregating author behavior, and multidimensional indexing of social media content. Finally, Mor noted that we’re moving from a world of email to a “social awareness stream”, in which the content we directed content at a group and have lower expectations of readership than email. As with all of the panels, there were countless questions from the moderator and audience, particularly about determining the truthfulness of social media content and delivering social content in an effective user interface.

The final conference session was a conference was a full-group discussion that dived into the various topics addressed throughout the day. But Gene Golovchinsky provided the “one more thing” at the end, showing us a glimpse of a faceted search interface to explore a Twitter stream. It was an elegant finish to a day filled with informative and engaging discussion, and I look forward to seeing many of the participants in the WSDM conference over the next few days.

Of course, I’ll cross-post here. I also encourage folks to follow the live tweet streams at #ssm2010 and #wsdm2010, as well as Gene and Jeremy’s posts at the FXPAL blog.

To those attending: see you all tomorrow through Saturday! To everyone else: I will try my best to communicate the substance and spirit of the conference.

But I’m pleased to report that Endeca now has is its own blog, aptly entitled Search Facets. I’m not usually a fan of corporate blogs, but I like the approach Endeca is taking to this one. The folks who have posted so far are Adam Ferrari (CTO), Vladimir Zelevinsky (Research Scientist), and Pete Bell (Co-Founder)–an indication that the blog will contain substance, rather than warmed-over press releases.

Indeed, the posts so far are nice and meaty. I particularly like Adam’s post about “Vertical stores for vertical web search?“–it’s nice to see read intelligent analysis from someone who understand the strengths of both MapReduce and column-oriented relational databases.

Anyway, I’m delighted that my former co-workers have taken to the blogosphere, and I look forward to reading what they have to say!

Last week, I had the good fortune to attend a presentation by John Wang, search architect at LinkedIn. You may have read my earlier posts about LinkedIn introducing faceted search and celebrating the interface from a user perspective. John’s presentation at the SDForum took a developer’s perspective, discussing the challenges of combining faceted search and social networking at scale.

John was kind enough to publish his slides, and I’ve embedded them above. Unfortunately, there’s no recording of the extensive Q&A (which included various attempts to get John to reveal the precise details of LinkedIn’s data volume), but the slides are quite meaty.

Personally, I learned two surprising things from the talk.

First, I was surprised that LinkedIn dismisses index/cache warming as “cheating”, instead computing almost everything in real time. Specifically, I would have expected LinkedIn to cache information like a user’s set of degree-two connections: these are expensive to compute at query time, especially when the social graph is distributed and sharded by user. I did ask John whether LinkedIn recomputes a user’s degree-two network during a session, and he admitted that LinkedIn is sensible enough to “cheat” and not perform this expensive but almost useless re-computation.

Second, I learned about reference search, a feature I may have missed because it is only available for premium LinkedIn accounts. It’s a nice feature, allowing you to search against company + date range pairs. People who are familiar with implementing faceted search may recognize the preservation of such associations between facet values as a gnarly implementation challenge.

All in all, it was a treat to get this look under the hood, as well as to finally meet John in person. I also ran into Gene Golovchinsky there–so much for my spending a few days on the west coast incognito!

In any case, I’m looking forward to seeing Gene, some of John’s colleagues, and many more interesting people at the Search and Social Media Workshop (SSM 2010) on Wednesday. My apologies to those who aren’t able to attend this oversubscribed event. I promise to blog about it!

It features a keynote from Jan Pedersen, Chief Scientist for Core Search at Microsoft, as well as an impressive set of posters and panel sessions. Other participants include:

• Sihem Amer-Yahia, Yahoo!
• Jon Elsas, CMU
• Gene Golovchinksky, FXPAL
• David Hendi, MySpace
• LiangJie Hong, Lehigh U.
• Jeremy Hylton, Google
• Matthew Hurst, Microsoft
• Hilary Mason, bit.ly
• Richard McCreadie, U. of Glasgow
• Mor Naaman, Rutgers U.
• Meena Nagarajan, Wright State U.
• Igor Perisic, LinkedIn
• Jeremy Pickens, FXPAL

There’s still time to register if you’re interested!

As my wife, daughter, and I were walking home from a playground, we noticed a large number of fire trucks congregating a block away from our house. A quick search on Twitter explained what was going on, particularly by pointing us to this post on Gothamist–which as of this writing seems to be the only reporting about this incident.

I think this example tells us a lot about the utility of real-time search. Most of us don’t need real-time search to tell us about the news in Haiti, since a critical mass of major news providers is covering the story around the clock. Where real-time search matters most is at the personal level–specifically, when our personal urgency to obtain information is higher than that of the general population. In such situations, we’re willing to accept less polished–and even risk less accurate–information, particularly if the alternative is to wait until if and when news providers cover the story. At least to some extent, urgency trumps authority.

Yes, there are other use cases for conversational media like Facebook and Twitter, such as sharing the experience of watching a live event, or simply chatting with friends and strangers about arbitrary topics. But I wouldn’t consider such use of these media to be search. Real-time search, in my view, is about helping users obtain the latest information available–in accordance with their personal needs. Twitter and Google served me well today, and I’m grateful that real-time search gave me real-time peace of mind.

Earlier this week, Peter Morville and Mark Burrell presented a UIE virtual seminar on “Leveraging Search & Discovery Patterns For Great Online Experiences“. It sold out! And I thought Pete Bell and I had done well with our seminar on faceted search!

But I’m hardly surprised. Although I wasn’t able to attend it myself, I gather from Twitter and the blogosphere that it was a great presentation. I enjoyed serving as a reviewer for Peter’s new book on Search Patterns, and I contributed a bit to Endeca’s UI Design Pattern Library while I was there and Mark’s team was developing it.

In reading reactions to the seminar, I was particularly intrigued by a post entitled “Search and Browse” by Livia Labate on her fantastically named blog, “I think, therefore IA“. She raised a question that I think needs to be asked more often: when is (or isn’t) faceted search appropriate?

Her conversation with readers in a comment thread offered some possible answers:

• Faceted search helps users who think in terms of attribute specifications as filtering criteria.
• Faceted search supports search by exclusion, as opposed to by discovery.
• Faceted search requires a set of useful facets that is neither too small nor too large.

I’d like to propose my own answers. Here are the conditions for which I see faceted search being most useful:

• Faceted search supports exploratory use cases, in contrast to known-item search. For known-item search, users are better served by a search box to specify an item by name, or a non-faceted hierarchy to locate it. In contrast, faceted search optimizes for cases where users are either unsure of what they want or of how to specify it.
• Faceted search helps users who need or want to learn about the search space as they execute the search process. Facets educate users about different ways to characterize items in a collection. If users do not need or want this education, they may be frustrated by an interface that makes them do more work.
• The search space is classified using accurate, understandable facets that relate to the users’ information needs. As I’ve discussed before, data quality is often the bottleneck in designing search interfaces. Offering users facets that are either unreliable or unrelated to their needs is worse than providing no facets at all.

Given the above criteria, it’s not surprising that faceted search has been a huge success in online retail: shopping is often an exploratory learning experience, and retailers tend to have good data.

But the success of faceted search in retail overshadows other domains where faceted search may be even more valuable. My favorite example is faceted people search, most recently demonstrated by LinkedIn. I would love to see other entities (locations, businesses, etc.) receive similar treatment, at least in contexts where exploration is a common use case.

I think Livia is right to be skeptical about any interface that introduces complexity–and facets do introduce complexity. I hope that my guidelines help answer her question as to when that complexity is worthwhile and perhaps even necessary to help users satisfy their information needs.

Weren’t we just talking about what’s different about mobile search use cases and about how to make web search more exploratory? I may be biased, but I think that Google’s recently launched “near me now” button is a step in the right direction (no pun intended!) on both of these fronts.

I’m curious to hear unbiased feedback from iPhone and Android users who have gotten to play with it.

For me, 2009 marked the end of a decade-long run at Endeca, where I focused on bringing HCIR to enterprises. I’m particularly proud of two professional accomplishments: writing a book on faceted search, and organizing the SIGIR 2009 Industry Track.

But past is prologue. I spent the last several weeks of 2009 as a Noogler, and I launch into 2010 living and breathing search on the open web.

What’s on my mind? Here are some top-of-mind questions to which I hope to have better answers by this time next year:

• Exploratory Search: how should we determine that users want a more exploratory search experience, rather than one that minimizes time to a best-effort result? How should we respond to queries that clearly don’t have a single best answers, such as queries of the form [category] or [category location]?

• Mobile Search: should it be just like non-mobile search with a few tweaks to accommodate the device form factor? Or does / should mobile search fundamentally change the way we interact with information?

• Real-Time Search: is it more than real-time indexing plus emphasizing recency as a query-independent relevance factor? What are the use cases, and how should we be addressing them?

• Social / Collaborative Search: should we be looking to microblogging or other social media signals to augment (or even supplant!) link-based citations as authority cues? Should we be supporting mediated search by linking people to people, rather than directly to information?

• Transparency: is it possible to offer more transparency in relevance ranking without losing ground in the battle against spam and black-hat SEO?

To be clear, these are simply the questions that are on my mind–I’m speaking as an individual and not as a Google employee. That said, a great thing about being at Google is that there are people working on all of these areas. So I expect 2010 to be an exciting year!

Curious to hear what problems are on other people’s minds as we enter the new year. Comment away!

committed to providing a framework for any company big or small, rich or poor to do a deal with us to get access to the Firehose in the same way we did deals with Google and Microsoft. We want everyone to have the opportunity — terms will vary based on a number of variables but we want a two-person startup in a  garage to have the same opportunity to build great things with the full feed that someone with a billion dollar market cap does. There are still a lot of details to be fleshed out and communicated, but this a top priority for us and we look forward to what types of companies and products get built on top of this unique and rich stream.

That and some other details, like raising the API rate limit from 150 requests per hour to 1500,  may well bring on what Marshall Kirkpatrick of ReadWriteWeb calls “Twitter 2.0“. But it was something else in Kirkpatrick’s write up that caught my attention–this quote from Wow.ly co-founder Kevin Marshall:

The more I do with and around social data, the less interested I seem to become in ‘realtime’ and the more interested I become in ‘over time.’ When I first started hacking on Twitter (and Facebook) apps, I was in love with the idea of parsing and analyzing data in real-time and I was very link/content focused. But the more I build and use these tools, the more I see the value in the history and the trails of the data set.

I couldn’t have said it better! Not that I haven’t tried: you look back at my post about Topsy, you’ll see where real-time and over time meet. Recency matters, but the signal is far too sparse without some way to aggregate and analyze over time.

I’m thrilled that Twitter plans to open up its platform in a way that could enable analysis over semantic, social, and temporal dimensions. Now I’m curious to see what that access will look like, and what everyone has been clamoring for that access will do with it.

I believe there’s a clearly established business case for faceted search inside the enterprise, for site search (especially for retail and media / publishing sites), even for vertical search on the open web. In all of these cases, relevance-ranked results are insufficient to meet a large subset of users’ more exploratory information needs, and HCIR approaches like faceted search are an easy sell.

But it seems much harder to make this case for general web search. The track record of startups in this space isn’t very encouraging. That could be because no one has done it right, but Clayton Christensen’s theory of disruptive innovation would suggest that a successful entrant wouldn’t have to have parity across the board, but would simply need to win on an underserved market segment. Perhaps the increasing use of faceted search for vertical search is how this process is playing out, and faceted search for general web search may end up being a slow agglomeration of verticals.

I’m curious if others have been pursuing efforts like Eric’s. Are the available APIs powerful enough to prototype your own faceted web search engine? If they aren’t, then is this a potential business opportunity for one of the major (or non-major) search engines to promote innovation by offering an open system? Or, if Yahoo! BOSS already offers such an open system, what should we make of the scale of its impact?

Long-time readers may recall my post about visual search startup Modista last November, or this guest post by one of its principals. Unfortunately, the story has a sad ending. I hope that both this technology and its developers find a good home.

One of my favorite resources for learning about user experience is the UXmatters blog. This group blog boasts a set of authors that represent a diverse collection of industry practitioners (and one academic) and offer concrete case studies and recommendations.

For example, in “Best Practices for Designing Faceted Search Filters“, Greg Nudelman offers a constructive critique of the Office Depot search user interface. Some of his material will be familiar to those who have read my faceted search book (particularly the chapter on front-end concerns), but the focus on a single example makes for a compelling read. I also liked Greg’s most recent post, entitled “Cameras, Music, and Mattresses: Designing Query Disambiguation Solutions for the Real World“. I was amused that he and I use the same “canonical” example for the need to offer clarification before refinement.

Here are a few more posts from other authors to give you a taste for the blog:

• “First, Do No Harm” by Pabini Gabriel-Petit
• “The Five Competencies of User Experience Design” by Steve Psomas
• “Beyond Usability: Designing Web Sites for Persuasion, Emotion, and Trust” by Eric Schaffer

If you are a user experience professional, in name or in deed, then you should be reading the the UXmatters blog — or perhaps even contributing to it. Of course, you’re always welcome to contribute a guest post here too.

LinkedIn started rolling out a beta version of faceted people search back in July. Now it’s officially out of beta, as announced on their blog. I’ve re-posted the video above in case you missed it in July.

Interestingly, LinkedIn developed its own tool to support the combination of faceted search with social network search: Bobo-Browse (Otis mentioned it in our recent presentation to the New York CTO Club). I helped develop similar functionality when I was at Endeca, so I know how hard this problem is. LinkedIn has done an impressive job–and has applied it to one of the most valuable data sets on the web. Bravo!

But I can’t help asking for just one more thing. LinkedIn has great semi-structured data about its 50+ million members. I’d love to be able to explore that data using more facets–in particular, facets relating to people’s job skills and expertise. I hope that’s something they’re working on. Perhaps a good topic of conversation at the upcoming Workshop on Search and Social Media!

Most karaoke venues provide printed song books, typically sorted by title and by artist. This approach is certainly adequate for very limited selections, but it doesn’t scale gracefully. Indeed, one of my favorite karaoke bars, the Courtside in Cambridge, MA, has a fantastic song selection that is only accessible through printed books. Kinda frustrating for a search guy, even though the staff is very helpful!

My regular karaoke venue in New York, Second on Second, is a bit more technologically advanced: it provides computers with dedicated software that allows patrons to search through their song catalog. Aside from being faster than thumbing through books, the software makes it possible to find songs when you only remember words that are in the middle of song or artist names.

But even such a system only addresses known-item search–in this case, looking for a song or artist by name when you know precisely what you are looking for. There’s room for incremental improvement here, e.g., searching for songs based on the lyrics you remember. For example, many people remember a famous David Bowie song based on its protagonist “Major Tom” rather than its title “Space Oddity“; fortunately, tools like Google’s music search are happy to make such connections.

But none of the karaoke search technology I’ve see to date supports exploration. Specifically, I’d love to go into a karaoke bar and have a procedure for finding songs I know that is better than trial and error. For example, I’d like to be able to see my options for hard rock 80s songs with male vocals. Or to find out which downtempo bands, if any, are on the menu. A little faceted search would go a long way towards making the song-finding experience more pleasant and efficient.

But why stop there? I’d really like a system that suggests songs based on what it knows about me. For example, knowing that I like to sing Scorpions songs is a reasonable basis to suggest similar artists like Def Leppard and Guns N’ Roses. Or perhaps to suggest 80s songs in general–after all, karaoke roulette notwithstanding, most people sing songs they know (or at least think they know), and their song knowledge tends to have some temporal locality. I’m sure you can imagine far more sophisticated personalization–and such personalization could be accomplished with complete transparency to the user.

Even if you aren’t into karaoke (and yet have managed to read this far!), I hope you can appreciate the universality of the information needs I’m describing. Exploratory search is everywhere. But I think it’s easiest to demonstrate its practical importance by working through concrete use cases. As an HCIR advocate, I’ve repeatedly learned the lesson that such demonstrations are critical in order to successfully evangelize this worldview.

Otis Gospodnetic and I recently gave a talk at the New York CTO Club on faceted search. The club is a group of senior technologists who meet monthly in midtown Manhattan to host breakfast presentations and to share ideas and expertise. Those of you who have read my book or attended the UIE virtual seminar a few months ago that I gave with Pete Bell (whom I worked with for 10 years at Endeca) might recognize some of my material. Otis focused on the specifics of implementing faceted search using the open-source Solr platform.

Here were the major take-aways:

• Think about what users are trying to do, not just how they search.
• Facets get polluted with bad result sets, so offer clarification before refinement.
• Don’t just move the information overload problem to the facets! Show less, not more.
• Look at the potential data facets you already have, you will be surprised.
• Facets can come from new data, e.g. sentiment.

• BIA / Kelsey Blog
  • By The Kelsey Group, a division of BIA Advisory Services that provides data and analysis on directories and local media.
• Exploring Local
  • By Mike Dobson, President of TeleMapics, a company that provides consulting services focused on local search.
• Local SEO Guide
  • Andrew Shotland‘s blog on local search optimization, small business marketing & search engine optimization strategy.
• Localsearchdatabase
  • By Gib Olander, Director of Business Development for Localeze, an online content management company serving businesses, local search engines and consumers.
• Mihmorandum
  • David Mihm‘s blog on local search engine optimization and marketing.
• Screenwerk
  • Greg Sterling‘s thoughts on online and offline media. Sterling used to run The Kelsey Group’s Interactive Local Media program.
• SEO Igloo Blog
  • By Solas Web Design, which specializes in web design and search engine optimization for small businesses.
• Understanding Google Maps & Local Search
  • By Mike Blumenthal, whose company offers consulting services and market research advice relating to maps and local search.

Not surprisingly, these blogs offers me a critical perspective on how Google and other search engines serve the local space.  Granted, everyone has their own motives–and it’s hard to avoid some tension in a space with the competitive dynamics of local search. But now that I’m no longer an outsider myself, I appreciate having others to help keep me honest as I work to make local search better for users and businesses.

But working on the inside has also help me appreciate what Bob Wyman tried to tell me months ago–that Google has no philosophical predilection towards black box approaches, but rather is only limited by what technology makes possible and what its engineers can implement. I’d qualify that slightly by saying that I perceive an additional constraint: Google does have a strong predilection towards data-driven decisions. Some folks have found that approach objectionable in the context of interface design.

Anyway, if you’re a regular here, then you’re probably predisposed towards HCIR and exploratory search. In that case, I’d like to take a moment to help you appreciate the challenge I face on a day-to-day basis.

Which one of these two statements do you most agree with?

1. We need better data quality in order to support richer search user interfaces.
2. Richer search user interfaces allow us to overcome data quality limitations.

On one hand, consider two search engines whose interfaces are designed to support exploratory search: Cuil and Kosmix. Sometimes they’re great, e.g., [michael jackson] on Cuil and [iraq] on Kosmix. But look what can happen for queries that are further out in the tail, e.g. [faceted search] on Cuil [real time search] on Kosmix. Yes, the kinds of queries I make. I don’t mean to knock these guys–they’re trying, and their efforts are admirable. Moreover, both generally return respectable search results on the first pages (in Kosmix’s case, through federation). But the search refinements can be way off, and that undermine the overall experience. I strongly suspect that the problem is one of data quality, along the lines of what others have argued.

On the other hand, some of the work that I did with colleagues at Endeca (e.g., work presented at HCIR 2008 on “Supporting Exploratory Search for the ACM Digital Library”) at least dangles the possibility that the second statement holds–namely, a richer user interface could help overcome data quality limitations. Interaction draws more of the information need out of the user, and the process may be able to mask imperfection in the data. For example, it’s clear to users–and clear from the search refinements–that [michael jackson beer] and [michael jackson -beer] are about different people. If we can just get that incremental information from the user, we don’t have to achieve perfection in named entity recognition and disambiguation.

I think there’s some truth in both arguments. Data quality is a major bottleneck for effectively delivering an exploratory search experience, and data quantity, much as it helps, is not a guarantee of quality. Richer interfaces offer the enticing possibility of leveraging human computation, but they also introduce the risk of disappointing and alienating users. Even for an HCIR zealot like me, the constraints of reality are sobering.

And yes, speed and computational cost matter too. But hey, it wouldn’t be a grand challenge if it were easy!

Google returns standard results for such searches:

Bing is generous to a fault, saving you a click if you choose to use one of its leading competitors:

Finally Yahoo, whose CEO claims “we have never been a search company,” seems quite eager to keep searchers from going elsewhere:

It’s easy to dismiss these queries as corner cases, but the logs show that they really happen. And, as browsers increasingly blur the line between an address bar and a search box, it’s not unreasonable to consider that switches between search engines are likely to commence with such queries.

Earlier this week, Marti Hearst gave a Tech Talk at Google about her recently published book, Search User Interfaces. Fortunately for those of us who missed (myself included!), it is now available on YouTube. Enjoy! (via Jon Elsas)

Consider MusicLab, an “experimental study of self-fulfilling prophecies in an artificial cultural market“. In this study, sociologists Matt Salganik, Peter Dodds, and Duncan Watts manipulated the social information available to consumers (specifically teens) regarding their peers’ musical tastes. The experimenters’ goal was to empirically validate a quantitative model of social contagion.

But we can look at this study another way: by isolating the social factors that influence musical taste, the experimenters were also isolating the non-social signal–in theory, how popular a song would be in the absence of social signaling. Indeed, they found that, if they measured a song’s quality by isolating out the social factor, “the best songs never do very badly, and the worst songs never do extremely well, but almost any other result is possible”.

It’s interesting–interesting to me, at least!–to ask if search engines can do the same for search. One of the frequent objections to link-based authority measures like PageRank is that they make the rich get richer. “Real-time” variants like re-tweet frequency (and even TunkRank) suffer from the same weakness. Unchecked, these measures can cause authority / influence market has to resemble a winner-take-all market.

It strikes me as interesting to learn from cases where searchers swim upstream against the social signals to find information. Of course, you may already see the contradiction–this is just another kind of social signaling! Still, it seems like it might be a way to hedge our bets and against the weaknesses of positive feedback and spammers. In a similar vein, we might look at how users find information that suffers from poor accessibility or retrievability.

I don’t have answers about how to pursue such an approach, or whether it would even be feasible to do so. But I hope you agree with me that it’s an interesting question.

Google’s recently released Image Swirl is slick. But I’ve been struggling to figure out whether it’s useful or simply a showcase for cool technology.

And that’s prompted me to think about the overloaded term “exploratory search“. A while back, I tried to define exploratory search based on what it is not. This time, let me aim to positively characterize what I see as its two primary use cases:

1. I know what I want, but I don’t know how to describe it.
2. I don’t know what I want, but I hope to figure it out once I see what’s out there.

The first use case cries out for tools that support query refinement or elaboration. Existing tools span a range from suggesting spelling corrections (aka “did you mean”) to offering semantically or statistically related searches that hopefully provide the user with at least a step in the right direction. One of my favorite approaches, faceted search, is primarily used to support query refinement through progressive narrowing of an initial search query.

The second “I don’t know what I want” use case is fuzzier. In the language of machine learning, this use case is unsupervised, while the previous one is supervised. In general, it’s a lot harder to define or evaluate outcomes for unsupervised scenarios. Indeed, Hal Daume has argued that we should only do unsupervised learning if we do not have a trustworthy automatic evaluation metric. That’s a strong position, and you can see some of the counterarguments in his comment thread. But, going back to our scenario, it’s really hard to judge the effectiveness of tools like similarity browsing when they support exploration in the absence of any concrete goal.

With that in mind, I’ll reserve judgment on the utility of tools like Image Swirl. To the extent that it aims at the first use case, clustering images for a particular search, I’m ambivalent. I’d prefer a more transparent interface, in which I have more of a sense of control over the navigational experience. I suspect it is more aimed at the second use case, offering a compact visualization of what is out there.

Besides, as some folks have brought up at the HCIR workshops, it’s important that we make information seeking fun. And Swirl certainly scores on that front.

So reads the first claim from a patent application that Apple recently filed (with Steve Jobs as first inventor, no less!) for technology to deliver a rather compelling ad-supported business model. Or perhaps the better word is compulsory. You can read an analysis by Randall Stross in the New York Times.

I agree with Stross that it’s hard to imagine Apple ever implementing the technology described by the patent application–indeed, Apple has been one of the few success stories for paid digital content models. That said, the approach does feel like at least one endpoint for the ad-supported model–it guarantees the advertisers the attention that they are paying for by subsidizing content or services.

The advertising business is a bit more top of mind for me, now that it pays my salary. Google’s approach, however, follows the aphorism that honey catches more flies than vinegar: it tries to target ads well enough that users want to click on them, rather than to simply endure them as a cost of subsidizing free services. Google’s revenue (and the popularity of PPC models in general) is a testament to the success of this approach, my occasional rant notwithstanding.

In general, the industry seems to have found a compromise in how aggressively to push ads at users. Users can safely ignore (or even block) sponsored links, but few people do.  Pre-roll ads on video sites (i.e., advertising before a video starts)  are more invasive, but a number of sites let users skip them. You can read why the YouTube folks are testing this approach. Advertisers–or at least ad-supported services–seem to recognize that they can’t cross the line between pursuing users’ attention and annoying users to the point of alienation.

Still, technology like Apple’s patent application describes shows that it is possible for the ad-supported model to take a more more aggressive approach. Part of me wonders if more aggressive ad-supported models would revitalize paid content models, as users would stop perceiving the former as free. But I suspect that the gentler ad-supported model is here to stay, and that it will continue to strive toward the point of optimal effectiveness.

The deadline for proposal submission is November 30th–you only have to submit a 250-word abstract.

Here is the call for proposals:

We seek dynamic speakers who can talk knowledgeably about detailed aspects of how to implement and maximize search within an organization. Search can no longer be viewed as a stand-alone application. It is increasingly part of everything we do and has become the de facto gateway to information in the enterprise. This year’s Summit will examine the ways to leverage search tools, information architecture, classification, and other strategies and technologies to deliver meaningful results—not just in terms of information, but to the bottom line.

Ours is a well-informed, tech-savvy audience, so proposals should be specific and detailed. Consider topic such as:

• Integrating search into enterprise systems and workflow
• Customizing your search solution/ Task-specific search
• Compliance, records management, and eDiscovery with effective search
• Migrating your search engine
• Social search and social tagging strategies & solutions
• Search-enabled decision making
• Business intelligence, data mining
• Search as the gateway to enterprise information
• Optimizing the interface and user experience
• Navigational tools—context, facets, entity extraction, clustering, and visualization
• Emerging trends, the future of search
• Overcoming information overload
• Categorization techniques
• Semantic Search
• Query Federation & Federated Search
• Enhancing an existing solution

If you represent a company that has an enterprise search software product, your best bet to be on our program is to collaborate with a customer to submit a case study to be presented by them, following the guidelines above.

If you need more information–or more time–I encourage you to reach out directly to Michelle Manafy, the conference chair.

As I finish my first week, I can sum the experience in a word:
overwhelming. The tools for accessing internal information are better
than I expected, but both the volume of baseline knowledge–technical
and cultural–and the relentlessness of the update stream are
daunting.

Indeed, the internal ecosystem is so rich that it’s easy to forget
there is a world outside it–ironic given Google’s enormous role in the world outside it! Then again, this is just my first week–it will take me
some time to pop up the stack from the build system to the surface.

Will I continue blogging at The Noisy Channel?

Absolutely! I’m committed to posting at least weekly, and I’ll try to do better than that once I’m settled into my new environment.

Will I participate in scholarly conferences and workshops?

Of course! I’m co-organizing SSM 2010, which will be held in conjunction with WSDM 2010 in February, and of course HCIR 2010, which will be held in conjunction with IIiX 2010 in August. You probably won’t see me at vendor fests, but I do hope to continue bringing industry practitioners and academic researchers together.

Will I blog about Google?

I certainly won’t disclose any confidential information–people get fired for that–or worse. And, given how much access I will have to such information, I will err on the side of caution, only discussing information that I’m sure Google has released to the general public. Beyond that, I’ll exercise common sense. I don’t want to either come across as a shill for my employer or to spar with my new colleagues in public. Subject to those constraints, however, I can and will blog about Google.

Can I get you a job at Google?

I can advise you and connect you to a recruiter, but that’s the limitof my power. The hiring process here is specifically designed to prevent any individual from manipulating it–even me!

Will I talk about what I’m working on?

See above regarding confidential information. I’ll be delighted talk about anything I’m working on that Google has decided to disclose publicly.

Does Google know about my karaoke habit?

Too late, they’ve already signed the offer letter.

This is my last week at Endeca. The decision to leave has been a heart-wrenching one: not only have the past ten years been the best of my life, but my experiences at Endeca have defined me professionally. Moreover, Endeca is riding a wave of success with recent advances in our products, new relationships with key partners, and fascinating new deployments.  (You can read Endeca’s latest announcements in our newsroom).

Ironically, it is this very success that compels me to move on. In the past several years, I have developed an increasing passion for search on the open web–an interest only furthered by the explosion of social media.

That is why I’ve decided to accept an opportunity at Google’s New York office. Readers here know that I’ve been a very public critic of Google’s simplistic approach to user interaction on the open web. I’m being offered an opportunity to help fix that approach–and it is an offer I can’t refuse. My mission is to apply my passion for human-computer information retrieval (HCIR), an approach that Endeca has pioneered in the enterprise, to the world’s largest information problems–and where better to do that than at the company that aspires to organize the world’s information.

This moment is bittersweet: I am excited about the new experiences that await me, but I have a heavy heart as I turn in my badge and part with a world-class team that has succeeded against incredible odds.

Given my role and tenure at Endeca, I want to say explicitly that this move is about my personal ambition. My passion for web search and social media, which have grown exponentially over the past couple of years, simply doesn’t align with Endeca’s focus in the enterprise.

Also, I want to make clear: Google hired me because of my values, and not in spite of them. I know that some folks will find it difficult to reconcile my criticisms of Google with my decision to join. That’s why there’s an opt-out village! Seriously, though, I take my values with me. Google is offering me the opportunity to channel my passion for HCIR into action, on the world’s largest stage. I’m well aware of the magnitude of the challenge, but hey, I’m feeling lucky.

In “Using Twitter Lists To Judge Influence“, Todd Zeigler of the Bivings Report writes:

I think Twitter Lists will end up helping separate the men from the boys when it comes to influence.  In addition to seeing a Twitter users follower count, we can now see the number of other Twitter users who have added them to lists (example to the right).  I would argue that getting added to a list is a bigger deal than simply getting someone to follow you.

I’m certainly intrigued by Twitter Lists, but I’m skeptical that counting how many lists someone is on will prove that much more useful than follower count. For example, I currently have 1159 followers, am on 33 lists, and have a TunkRank of 24.1. For grins, here’s a handful of people who have similar stats:

• Evan Sandhaus: 796 followers, 21 lists, TunkRank = 17.2
• Josh Young: 801 followers, 25 lists, TunkRank = 14.3
• Chris Ahearn: 1108 followers, 14 lists, TunkRank = 30.1
• Brynn Evans: 1303 followers, 33 lists, TunkRank = 18.9
• Eric Andersen: 1543 followers, 37 lists, TunkRank = 3.1

While I can’t generalize from a few arbitrarily selected data points (though Gladwell seems to have no trouble doing so in Outliers), my suspicion is that list count will be highly correlated to follower count–and may actually be a noisier signal because the numbers are so much smaller.

Of course, there’s no reason we should use raw list counts–any more than we should use raw follower counts. Just as TunkRank aspires to model attention scarcity and recognizes that not all followers are created equal, an effective measure of how lists contribute to influence must recognize that not all list memberships are created equal either.

I’ve been chatting with Chris Langreiter, who is working on enhancements to TunkRank to address some of the oversimplifications of its model, as well as with Jonathan Glick and Ken Reisman at TLists. I’d like to see online influence–on Twitter and in general–measured more effectively. It will be great if lists can help, but we can’t make the same naive mistakes as those who were quick to embrace follower count as a measure of authority.

With all of the activity around e-books last week, you might think that the online world wasn’t paying attention to the media category most transformed by the Internet music. But a week is a lifetime in the ADD-addled technology press, and today’s top story is that Google is “making search more musical“. From the official blog post:

Now, when you enter a music-related query — like the name of a song, artist or album — your search results will include links to an audio preview of those songs provided by our music search partners MySpace (which just acquired iLike) or Lala. When you click the result you’ll be able to listen to an audio preview of the song directly from one of those partners.

As with most Google features, this one is being rolled out gradually. If you’re impatient (like me), you can try it directly from this page. Or you can watch the video above.

My first impression: this is great feature to improve known-item search, and it’s nice that they’ve partnered with folks that often let you hear whole songs, rather than 30-second snippets. The selection seems limited, but it could be that my tastes are a bit obscure. I’m curious if others share my sense that the catalog is much smaller than the ones on iTunes or Amazon.

But, as music IR specialist and fellow HCIR advocate Jeremy Pickens points out, Google is “doing to music what they did to the web“. I’m not as concerned as Jeremy is about the prospect of musical tastes being homogenized through the “rich get richer” effect of ranking–perhaps because we’re already there. Not only is pop music self-perpetuating (see this great study by my friend (and Princeton sociologist) Matt Salganik and his former advisor Duncan Watts), but even recommendation engines quash diversity. Google really can’t make things that much worse.

Besides, much as Google’s default search leads many searchers to Wikipedia, a great starting point for exploratory search, the new music search leads users to Pandora, which is probably the leading engine for exploratory music search offers users a more exploratory user experience (though it would be great if they also linked to last.fm) (thanks Jeremy!). OK, maybe “leads” is a strong word for a “listen on” link below the search result, but it’s there for people in the know.

I’d love to see Google embrace HCIR. But I appreciate the improvements to known-item search too, especially if they can delegate the HCIR functionality to others that focus on it.

The slides for Ben Shneiderman‘s HCIR 2009 keynote on “The Future of Information Discovery” are now available on the workshop web site. I’ve also taken the liberty to upload them to SlideShare and embed them here. The slides don’t do justice to Ben’s presentation style, but hopefully they at least communicate a taste of the material he covered and his vision of where HCIR needs to go as a field and community.

Google may be an also-ran in the social networking market with its Brazil-centric Orkut service, but that hasn’t stopped the search giant from adding social features to its products. A post at the (unofficial) Google Operating System blog recounts the history of Google Reader’s social evolution, up to but not including its latest update last week. SearchWiki, though not a social search feature per se, allows users to share personal annotations of their search results, as does the more recently introduced Sidewiki. And, like Bing, Google has established a partnership with Twitter in order to surface “social” results.

But the feature announced today, which Google is actually calling “Social Search“, is a much bigger step, even if it is tucked away as an experiment on Google Labs. From the official blog post:

With Social Search, Google finds relevant public content from your friends and contacts and highlights it for you at the bottom of your search results. When I do a simple query for [new york], Google Social Search includes my friend’s blog on the results page under the heading “Results from people in your social circle for New York.” I can also filter my results to see only content from my social circle by clicking “Show options” on the results page and clicking “Social.”

I gave it a whirl, search for “noisy channel” and then restricting the search to content from what Google considers my social circle. The results are as promised, and could further refine to results by author name, selecting from a familiar list of Neal Richter, Jason Adams, Daniel Lemire. Ken Ellis, and Joshua Young (though for some reason Josh’s link didn’t work). Cool! Except that there are a lot of names missing (check out the bloggers in The Noisy Community) and, more importantly, I can’t further refine or even sort the search results. Indeed, the ordering of search results seems quite arbitrary–a phenomenon I’ve noticed more generally for search engine ranking of social media content.

In short, Google Social Search is a welcome initiative, but there’s a lot more work to do before I would find a productive use for it. Given the mismatch between social search and black-box relevance ranking, a little bit of HCIR would go a long way towards making this feature practically useful.

But this year was different. We were overloaded with strong submissions from all corners, and we had to turn people away for lack of capacity! While we didn’t relish saying no to prospective participants, these are great problems to have! And, thanks to Nick Belkin and Diane Kelly, we’ve arranged to greatly increase that capacity at HCIR 2010–more on that in a moment.

Max Wilson has already written up an excellent summary of the workshop, which I encourage you to read. You can also see the live tweet stream at #hcir09. Rather than duplicate these efforts, let me add my personal reflections as an organizer and participant.

Ben Shneiderman‘s keynote address was sweeping and inspiring. I expected him to talk about information visualization, the area where he is most known for his contributions. He did present some examples of his group’s work on visualization-centric interfaces to support medical research, but his overall presentation took the much more ambitious approach of discussing the past, present, and possible future of HCIR. Specifically, he urged us to link our work to societal goals, such as the United Nations Millennium Development Goals. His challenge may seem impossibly idealistic, but I agree with his assertion that it is a practical one: we will do our best research by grounding ourselves firmly in the real and pressing problems of our age. Last year’s keynote speaker went on to win the Gerard Salton Award; I can only hope that Ben receives comparable accolades for his past accomplishments and future contributions to HCIR.

A new feature for this year’s workshop was having a “poster boaster” session, in which each of the presenters in the poster session had one minute to pitch his or her work.  For those of you unfamiliar with this format, I highly recommend it. The compressed format forces presenters to distill the essence of their contributions–a useful exercise in general. And the audience doesn’t get bored: if you decide halfway into a presentation that you aren’t interested, then you only have to wait 30 seconds until the next one! Not the we had that problem: the posters were consistently interesting, as the submissions were unusually strong this year. You can download the full workshop proceedings here.

Even the full presentations weren’t that long. The five speakers were each allotted ten minutes, with a healthy amount of time reserved for a panel-style Q&A sessions. The papers in this session were, by design, some of the more controversial ones. In particular, Ellen Voorhees delivered a full-throated defense of Cranfield / TREC-style evaluation: “I Come Not to Bury Cranfield, but to Praise It” (similar to her presentation at the 2006 Workshop on Adaptive Information Retrieval that I discussed on this blog last year). Her reminder of HCIR’s challenges on the evaluation front surely ruffled some feathers, but all of us HCIR avocates need to address these challenges if we want researchers (and practitioners) outside our community to drink our kool-aid.

The above format was already quite interactive (as befits a workshop about interaction), but the second half of the day was explicitly designed to facilitate discussion. We had lunch on site, followed by a one-hour poster session.  We then had two one-hour guided discussion sessions to address the theoretical and practical concerns of HCIR. As organizers, we seeded both sessions with questions, but we also incorporated concerns that had come up during earlier discussions.

Finally, I am grateful to our sponsors. Catholic University was a gracious host and sponsor, providing the workshop with a great space and very helpful student volunteers. Between that and the financial contributions of Endeca and Microsoft Research, we were able to continue our tradition of not charging attendees for the workshop. I can’t promise that will continue indefinitely, but I am glad that our insistence on emphasizing substance over frivolous amenities has helped us deliver what I believe to be some of the best bang-for-buck in the scholarly community.

I’m already excited about HCIR 2010. Unlike the past three workshops, which have been held as independent events, next year’s workshop will be co-located with the Information Interaction in Context Symposium (IIiX’10) in New Brunswick, New Jersey. The workshop will take place on August 22nd, breaking our unintended tradition of holding the workshop on October 23rd. Nick Belkin assures us that there will be lots of space, so hopefully we’ll be able to accommodate everyone who is interested. We’ll also be soliciting sponsors for both the workshop and the broader symposium.

But there’s more to HCIR than enjoying each other’s company at workshops. We must spend the remaining 364 days of the year fleshing out our vision, and relating that vision not only to the disciplines HCIR explicitly integrates, but to pressing social concerns. It is up to us all to make our work relevant.

When my daughter was born almost two years ago, I wondered if she’d grow up reading books. After all, I do most of my reading online, and increasingly find myself reading short articles rather than whole books. Needless to say, she’s loved books so far, even if she’s shredded a few.

But the bigger surprise for me is that books–specifically e-books–have become such a hot industry. When I briefly worked for a consulting firm after grad school in 1999, my first assignment was to evaluate the e-book market. The readers then consisted of the Rocket ebook and SoftBook Reader. Needless to say, I correctly predicted at the time that the ebook-market wasn’t ready for prime time.

But fast forward to the present. Amazon has given the e-book market some credibility: Citigroup says they sold 500K Kindles in 2008, and Forrester predicted they will sell 1.8M units this year.

But the last days (and even the last 24 hours!) of news show that the e-book market is only starting to open up:

• In May, Sony, whose e-reader sales have lagged behind the Kindle, announced a partnership with Google in May in order to make copyright-free books available for free.
• Google just announced a service called Editions that it plans to launch in 2010 (by when it will have presumably settled the Google Books Settlement Agreement).
• The Internet Archive just announced the Bookserver project as “a growing open architecture for vending and lending digital books over the Internet”.
• Spring Design just announced Alex, an e-book reader based on Google’s Android operating system.
• Barnes & Noble is expected to announce an e-reader that competes directly with the Kindle and has generated lots of buzz through leaked photos.

I grew up on books, and I’m excited to see that, a decade after the initial market failures, e-books (like touchscreens) are a mainstream reality. I still worry about who will buy them, especially considering that the marginal cost of distributing a typical e-book is even less than that of distributing a 5-minute song. A quick scan of a popular file-sharing site reveals that the pdf version of bestseller The Lost Symbol takes up less than 3MB.

Still, I’ll take a moment to celebrate the progress of technology. I’ve always known that reading was cool, but now we have the gadgets to prove it!

Who will buy this wonderful morning?
Such a sky you never did see!
Who will tie it up with a ribbon
And put it in a box for me?

Of course, the trope that the best things in life are free predates musical theater, let alone the web. But recent years have witnessed dramatic changes in our price sensitivities in every genre of digital (or digitizable) content, and I’m curious (sometimes morbidly so) about where it goes from here.

I won’t make you suffer through a rant about the malaise of the music and news industries–those topics, important as they are, have been overplayed in the blogosphere. If you need a refresher, I suggest Lawrence Lessig and the Nieman Journalism Lab as some of the more rational voices contributing to the discussion.

But it’s not just news and music that are experiencing the effects of the “information wants to be free” movement. Consider these industries:

• Books. Many publishers worry that the Kindle has been setting a consumer expectation that a book should only cost $10. Indeed, a recent price war between Amazon and Wal-Mart drove some of those prices down to $8.99. Is this a boon for consumers, or a body blow to the publishing industry? It’s easy to evoke the $0.99 / per song expectation set by iTunes–but that change was more about disaggregating albums than about changing the per-unit cost. Besides, books have not yet had to confront the scale of unauthorized distribution that we see in the music industry. Legal or not, free is a potent source of price pressure.
• Software. Wolfram Alpha just made headlines by releasing a $50 iPhone app. Many have reacted that such a high price is outrageous and will doom the application to failure. They may be right on that latter point–the market will vote with its clicks soon enough. But I’m old enough to remember $50 as being in the ballpark of what it cost to purchase a new consumer software application. Even then, unauthorized distribution was an issue–remember the “don’t copy that floppy” campaign? Today, my impression is that few people consciously purchase consumer software–a trend that I at least date to Microsoft’s strategy of bundling its software into PC purchases. The most noted exceptions are console games (which are impressive holdouts in the consumer software space) and iPhone apps–with the caveat that only a tiny minority of apps make enough money for the creators to live on. (Update: just saw this note about how EA Sports President Peter Moore sees the current console game business model of cartridges and discs as a “burning platform”.)
• Television. Between Boxee and Netflix, there is a real chance that digital content’s cash cow, cable television, will see its regional monopolies disrupted. I can’t imagine that anyone will shed a tear for the cable companies. And yet I can’t help but wonder what happens as the notion of premium content is subsumed by an expectation that video content should be free. Are we heading towards a proliferation of cheaply produced reality TV, contests, and game shows–all sponsored by rampant product placement?

If we are to believe Mike Masnick, then the price of content is driven to its marginal cost. It’s pretty clear that the marginal cost of distributing most digital content is, while not free, close enough to be a rounding error. Should we be looking forward to a world where no one can charge consumers for content? Folks like Jeff Jarvis and Chris Anderson are cheerleading such a world as not only inevitable but a good thing–though both of them have had the sense to make some money on non-free books while the going is good.

Yes, there are and will always be business models to support content creators. In particular, one-time content (live events, consulting services) has some degree of insulation from the inexorable trend toward free. But what an inefficient turn of events, if people are rewarded for creating one-time content but not for creating far more valuable content that is useful to a broad audience of consumers!

I know that there are non-financial incentives that drive scholars, open-source developers, and activists to create free content. Indeed, I personally write this blog without any direct financial incentive. Perhaps these incentives will be the driving forces for content creation in the 21st century. One way or another, I hope we find a way to fund the things we value, rather than devolving into a locally optimal rut where value creation isn’t economic for the creators.

p.s. You can find the lyrics to Oliver for free online, and you can easily view an free (unauthorized) copy of a performance of “Who Will Buy?” on YouTube. Or you can buy the song for $0.99.

Here’s the key information from the workshop web site:

Overview

Social applications are the fastest growing segment of the web. They establish new forums for content creation, allow people to connect to each other and share information, and permit novel applications at the intersection of people and information. However, to date, social media has been primarily popular for connecting people, not for finding information. While there has been progress on searching particular kinds of social media, such as blogs, search in others (e.g., Facebook, Myspace, of flickr) are not as well understood.

The purpose of the 3rd Annual Workshop on Search in Social Media (SSM 2010), is to bring together information retrieval and social media researchers to consider the following questions: How should we search in social media? What are the needs of users, and models of those needs, specific to social media search? What models make the most sense? How does search interact with existing uses of social media? How can social media search complement traditional web search?  What new search paradigms for information finding can be facilitated by social media?

SSM 2010 follows up on the highly successful SSM 2009 and SSM 2008 workshops held at SIGIR 2009 and CIKM 2008 respectively. We are looking forward to an equally exciting workshop at WSDM 2010 in New York!

Format and Topics

We are planning for a full-day workshop consisting of invited speakers, organized in both plenary and panel sessions, and a contributed poster/demo session.

We solicit short (under 2 pages) position papers, posters or demo proposals to be presented as part of a poster session, describing late-breaking and novel research results or demonstrations of prototypes or working systems. All topics at the intersection of information finding and social media are of interest, including, but not limited to:

• Searching blogs, tweets, and other textual social media.
• Searching within social networks, including expert finding.
• Searching Wikipedia discussions and revision histories.
• Searching online discussions, mailing lists, forums, and community question answering sites.
• The role of human-powered and community question answering.
• Novel models of information finding and new search applications for social media.
• The role of timeliness, authority, and accuracy in social media search.
• Interaction between traditional web search and social media search.
• User needs assessments and task analysis for social media search.
• Interactions between searching and browsing in social media.
• Searching and exploiting folksonomies, tags, and tagged data.
• Spam and adversarial interactions in social media.

Ideal papers may include late-breaking and novel research results, position and vision papers discussing the role of search in social media, and demonstrations of prototypes or working systems. Note that the workshop proceedings will not be archived or considered as formal publication, to encourage the informal atmosphere and to allow the authors to publish expanded versions of the work elsewhere.

The poster/demo proposals should be in standard ACM SIG format, more details to be posted soon.

Submissions are due on December 15th. I hope to see some of you there! Meanwhile, feel free to suggest ideas for invited speakers who have done interesting work at the intersection of social media and search, and I’ll share your suggestions with my co-organizers.

But apparently the Huffington Post is blazing new trails in this area. The Nieman Journalism Lab reports that:

The Huffington Post applies A/B testing to some of its headlines. Readers are randomly shown one of two headlines for the same story. After five minutes, which is enough time for such a high-traffic site, the version with the most clicks becomes the wood that everyone sees.

NJL also reports that Huffington Post social media editor–and long-time Noisy Channel reader–Josh Young uses Twitter to help crowd-source better headlines.

I’m sure this approach must rattle some old-school journalists. And there is a real danger of optimizing for the wrong outcome. For example, including the word “sex” in this message might improve its traffic (the popularity of this post attests to that), but to what end?

Still, I don’t see this use of technology as cramping anyone’s style. Most of us write to be read–especially those in the media industry who are trying to monetize their audiences. Measurable success matters, and there’s no harm in trying to maximize it.