It’s that time — the end of another great year. It’s been a phenomenal year for LinkedIn, for my amazing team of data scientists, and for me personally.

The end of year is also an exciting time of transition. I enter 2013 thinking about my daughter starting kindergarten, the house awaiting me after a lifetime of apartment dwelling, and of course all the great things my team is setting out to accomplish.

One of those transitions concerns this blog. About a month ago, I started posting on LinkedIn as an “influencer” — one of a couple of hundred people privileged to use LinkedIn as a native publishing platform. The thousands of people following me there and engaging with my content have convinced me to go all in.

I will keep this site up with the hundreds of posts I’ve published over the past 4+ years, but I’ll put The Noisy Channel on mute for the foreseeable future. So, whether you’re a long-time reader or someone who just got here, I hope you’ll follow me on LinkedIn and read my posts there.

I’m excited about the upcoming year. There’s lots of great stuff on my team’s roadmap, as well as fun opportunities to participate in conferences. Hopefully I’ll see some of you at the Stanford Conference on Computational Social Science and at the O’Reilly Strata Conference. But most of all I hope to continue engaging with all of you offline and online.

Happy New Year, and see you on LinkedIn!

What a wonderful week! On Tuesday, my daughter Lily turned 5. We celebrated by taking her out to her first karaoke night, where she had a blast (check her out singing Kimbra’s part in “Somebody That I Used To Know“) as did we! I’ll post a video soon. And no, she didn’t sing in a bear outfit — but she did dance in that costume.

This same week, I’m celebrating my second anniversary at LinkedIn. It’s amazing how quickly two years have gone by, and how the team has grown in numbers and accomplishments. My team post from May already feels so dated! I’ll update it in the next few weeks.

And I just passed 5,000 followers on LinkedIn. I’m gratified that people enjoy my writing, and also thrilled to be playing a part in LinkedIn’s content ecosystem.

I hope all of you are celebrating your own milestones as we approach the holidays and the end of 2012.

• Understanding Relevance
• Relevance is Complicated

I’ll continue to use The Noisy Channel as a personal blog, but I’ll be focusing my publishing efforts on LinkedIn. Hope you’ll follow me there!

My first two posts as an “influencer” are a bit meta:

• Influence and the Attention Economy
• Measuring Influence

I’m curious to hear what you think — both of the content and of the platform. I have no immediate intention to migrate the The Noisy Channel to LinkedIn, but I’m very excited by the initial feedback I’ve gotten for these posts.

Meanwhile, I encourage you to follow me on LinkedIn!

The moment I learned that CIKM 2012 would be held in Maui, I knew I had to be there. Having co-organized the CIKM 2011 industry event, I had enough karma to be invited as part of this year’s industry event, representing LinkedIn.

PRE-CONFERENCE: PSEUNAMI WARNING AND A WORKSHOP

But ironically in this workshop — and throughout the conference — there was more use of Twitter data than of Twitter itself. Most of the tweets using the #cikm2012 hashtag were my own.

DAY 1: USER ENGAGEMENT, EVALUATION BIAS

Tuesday opened with a welcome that included statistics showing how far CIKM has come as a top-tier international conference. There were 1,088 submissions this year! But the highlight of the opening was program co-chair Guy Lebanon demoing software to “improve” paper reviews. It was hilarious, if a bit close to home: the automatically generated reviews looked a lot like those generated by allegedly human reviewers.

We then proceeded to a keynote by Yahoo! Research VP Ricardo Baeza-Yates entitled “User Engagement: The Network Effect Matters!” The title was a bit confusing: it wasn’t about the conventional “network effect“, but rather about user engagement across a network of sites like those owned by Yahoo. He talked about different ways to measure user engagement, and noted that off-site (or, rather, off-network) links ultimately improve users’ downstream engagement. He also observed that style attributes outperform content attributes as predictors of user engagement. Lots of fascinating observations, but I’m curious how well they generalize beyond Yahoo.

I spent the rest of the day making tough choices among the various parallel sessions, starting with the morning session on information retrieval evaluation. Some nuggets from that session: captions and other surface features introduce significant evaluation bias; assessors have poor agreement when evaluating relevance in eDiscovery contexts; and system evaluation improves when it models user differences.

After lunch, I attended a session on web search. Some themes from that session: neighborhood-based methods are effective, whether the neighborhoods are based on document or user similarity; entities and structure are increasingly important for web search. After the coffee break, I went to the social networks session. Topics there included social contagion, online question answering, and social network data anonymization. The talks wrapped up just in time for us to watch the daily cliff diving ceremony before heading to the poster session.

DAY 2: QUERY PERFORMANCE PREDICTION, ABANDONMENT

Wednesday opened with a keynote by CMU professor Wllliam Cohen on “Learning Similarity Measures based on Random Walks in Graphs“. He described the framework and techniques that he and his colleagues used to build NELL (“Never-Ending Language Learning”). The keynote was pretty dense, but there are lots of papers available on the NELL publications page.

Then back to choosing among parallel sessions. Although I was tempted by the recommender systems session featuring presentations by my LinkedIn colleagues Mitul Tiwari and Bee-Chung Chen, I instead attended the session on ads and products. Two take-aways from that session: ad targeting benefits from explicit identification of user interests; influence maximization can be modeled adversarially as a two-player game.

After lunch, I attended the session on formal retrieval models and learning to rank. I most enjoyed the two talks by Oren Kurland that focused on query performance prediction. In particular, he offered a comprehensive probabilistic prediction framework that unifies most of the previously proposed prediction methods using a common formal basis. The session also included a deep dive into aspects of the IBM Watson question-answering system.

After the coffee break, I headed to another session on web search – one of my favorite sessions of the conference. There was a talk on query segmentation, a topic responsible for my most popular blog post. Also a great talk on identifying good abandonment, a problem I’ve been interesting ever since hearing about it at SIGIR 2010. Another talk about learning from search logs: generalizing from click entropy to “click pattern entropy” to analyze query ambiguity. And a talk on modeling domain-dependent query reformulation as machine translation using a pseudo-parallel corpus. All in all, a great session packed with practical content.

Then came a purely social evening. The conference reception was a luau, complete with kalua pig, mai tais, hula, and of course poi. Certainly my most memorable conference banquet. I didn’t take pictures, but I recommend Craig Stanfill‘s photos on Flickr.

And then some of us went to downtown Lahaina to see how the locals (and tourists) celebrate Halloween. Much as I missed spending Halloween with my family, I had a blast!

DAY 3: INDUSTRY EVENT

Thursday began with the last conference keynote: University of Kansas provost Jeffrey Vitter on “Compressed Data Structures with Relevance“. Like the previous keynotes, it was fairly dense, and I suggest you read the papers cited in his abstract if you’re interested in the technical details of how to search for query patterns in massive document collections.

Then came my main reason for attending the conference: the industry event. As seems to have become a pattern at information retrieval conferences, the industry event dominated the other parallel sessions, drawing a standing-room only crowd.

The event started with eBay VP of Research Eric Brill talking about “Having A Great Career in Research”. Unusual in a conference talk, he offered personal and practical advice to students on how to focus their passion and effort towards a happy and successful career. It reminded me of my blog post about dream, fit, and passion, and I hope students took it to heart.

IBM researcher David Carmel gave a talk entitled “Is This Entity Relevant to Your Needs?”. Noting that 71% of web search queries contain named entities (people, places, organizations), he advocated a probabilistic ranking approach to entity-oriented search that ranks retrieved entities according to amount and quality of supporting evidence.

Microsoft Technical Fellow (and former Yahoo! Fellow) Raghu Ramakrishnan talked about “The Future of Information Discovery and Search: Content Optimization, Interactivity, Semantics, and Social Networks”. He packed in a lot of nice material, most of which was from his tenure at Yahoo. He included a nice explanation of explore/exploit, which was also a reminder of how lucky we are at LinkedIn to have hired his former Yahoo colleague Deepak Agarwal.

After lunch, WalmartLabs Chief Scientist AnHai Doan gave a talk entitled “Social Media, Data Integration, and Human Computation”, in which he described constructing a “social genome” by mining social data, connecting it to web data, representing the combined information in a knowledge base. If you’re interested in more details, he’ll be giving an extended version of that talk at LinkedIn on November 29th!

Tencent Research Director Chao Liu talked about “Question Answering through Tencent Open Platform”. Beyond giving a great overview of one of the world’s largest internet platforms, he delivered great self-deprecating lines like “The name is ten cents, and the search engine is soso“.

I spoke next about LinkedIn‘s “Data By The People, For The People“. Given that the talk was right after Halloween and just before the presidential elections, I thought it appropriate to choose a title that would have appealed to one of America’s most distinguished presidents and vampire hunters. If you’re curious to learn more about data science and engineering at LinkedIn (including the publications I cited in my talk), check out http://data.linkedin.com/.

Groupon Director of Research Rajesh Parekh talked about “Leveraging Data to Power Local Commerce”. He focused on a key problem Groupon faces: determining and optimal category mix for each local market. He described how they approach this problem using portfolio theory.

After a coffee break, Adobe Chief Software Architect Tom Malloy talked about “Revolutionizing Digital Marketing with Big Data Analytics”. Apache Pig co-creator — and now Google researcher — Christopher Olston talked about work he did at Yahoo on “Programming and Debugging Large-Scale Data Processing Workflows”. Finally, Microsoft Distinguished Engineer Xuedong (“XD”) Huang gave a talk entitled “From HyperText to HyperTEC”, in which he woke up the audience by having us all participate in the “Bing it On” challenge.

FINAL THOUGHTS

All in all, CIKM 2012 was a great conference in an idyllic setting. Holding the conference in Maui might have been a bit distracting, but the desirability of the location also ensure a high-quality program.

My main complaint is that I don’t like parallel sessions — especially when the topics overlap significantly (e.g., web search sessions competed with those on ranking and recommendations). I’m also not convinced that talks have to be 25 minutes long. Perhaps the conference could more to a format of shorter talks and at least reduce the number of parallel sessions. It would also be great to see more opportunity for interaction — the coffee breaks always felt too short. For more of my thoughts on reforming academic conferences, see my 2009 blog post on the subject.

I also wish more attendees would embrace social media. It’s ironic that researchers who depend so heavily on social media data (especially Twitter) don’t engage in it personally. While I’m honored to have been the conference’s unofficial tweeter (see this visualization of the #cikm2012 tweets), I would have liked to see more attendees engage in a public online conversation. Hopefully others will at least blog about the conference.

Last year, I had the pleasure of co-organizing the CIKM 2011 Industry Event in Glasgow. This year, I’m honored to be part of the CIKM 2012 Industry Event program, along with top industry researchers from Adobe, eBay, Google, Groupon, IBM, Microsoft, Tencent, and Walmart Labs. I’ll be giving a talk on “Data By The People, For The People“.

I’m also thrilled to be joined by my colleague Mitul Tiwari, who will be presenting a paper on “Metaphor: A System for Related Search Recommendations“, work that he did with Azarias Reda, Yubin Park, Christian Posse, and Sam Shah.

Finally, we’ll be representing Manuel Gomez-Rodriguez and Monica Rogati at the poster session, presenting their work on “Bridging Offline and Online Social Graph Dynamics“.

If you’re attending CIKM, please make sure to say hi to me and Mitul. We’d be delighted to talk to you about the work that we and our colleagues are doing. Our data team also has a site showcasing our team, our publications, and some of the projects we can discuss publicly. And, of course, we’re hiring!

Human-computer information retrieval (HCIR) is the study of information retrieval techniques that integrate human intelligence and algorithmic search to help people explore, understand, and use information. Since 2007, the HCIR Symposium (previously known as the HCIR Workshop) has provided a venue for the theoretical and practical study of HCIR. We even inspired an EuroHCIR workshop across the pond that started in 2011 and is going strong.

Overview

The Sixth Symposium on Human-Computer Interaction and Information Retrieval (HCIR 2012) took place on October 4th and 5th at IBM Research in Cambridge, MA. The 75 attendees represented a cross-section of HCIR research and practice. Over a third of the attendees were from industry — including startups and large technology firms. We had a similar diversity of sponsors, benefiting from the generosity of FXPAL, IBM Research, LinkedIn, Mendeley, Microsoft Research, MIT CSAIL, and Oracle. And we had participants from 6 countries: Canada, Germany, Israel, New Zealand, Switzerland, and the United States.

Keynote

We started the Symposium with a keynote from UC Berkeley professor Marti Hearst, a pioneer in the area of search user interfaces, as well as a prominent researcher of information visualization, natural language processing, and social media analysis. Marti set the tone for the symposium with a visionary keynote that she entitled her “Halloween Cauldron of Ideas for Research”.

She started by talking about the unaddressed seams of sensemaking, reminding us that information seeking is only one part of an overall sensemaking process. She used the challenge of saving and personally organizing search results as an example of a neglected but crucial part of a search interface.

She then challenged us to think about how audio could be used in search interfaces. She cited a study showing that programmers comment their code better when the commenting interface uses speech rather than the keyboard. She then challenged us to consider how auditory notification or feedback could enhance the search experience.

Finally, she presented the idea of “radical collaboration”, offering as an example the use of Mechanical Turk to crowdsource vacation planning. The plans were tested by real tourists, who were delighted with the results.

Marti’s keynote was not only insightful and entertaining (one of her slides featured brain cupcakes!), but notable in how much she engaged all of us in discussion throughout her presentation. This approach was especially appropriate for an HCIR Symposium, given our emphasis on human interaction. For more detail about the keynote, I recommend Gene Golovchinsky’s summary.

Short Paper Presentations

After a coffee break, we had a session devoted to 5 short papers. Each presenter had 10 minutes: 5 minutes to present and 5 minutes for discussion.

• We started off with UXLabs director Tony Russell-Rose presenting “Designing for Consumer Search Behaviour“, joint work with University College London researcher Stephann Makri. Tony could not attend in person, so he submitted a video. He presented a framework for describing consumer search behavior along with concrete examples — many of them familiar from the time that Tony and I both worked at Endeca. Most of all, he emphasized the need to close the gap between information science research and industry practice.
• Then MIT professor (and Haystack principal investigator) David Karger talked about “Standards Opportunities around Data-Bearing Web Pages“. He argued that there is a small set of standard user interface patterns for authoring structured data: text search, sorting by properties, presenting items in a template, and faceted browsing. He then advocated that these primitives (which have already been implemented in the popular Exhibit framework) be incorporated into a W3C standard so that content authors can use them with the expectation that all modern browsers support them.
• Next, Harvard student Elena Agapie presented joint work that she did at FXPAL with Gene Golovchinsky and Pernilla Qvarfordt, entitled “Encouraging Behavior: A Foray into Persuasive Computing“. Information retrieval researchers and practitioners have often argued that longer queries lead to better retrieval performance. But how do we get users to enter longer queries. Elena and colleagues found that the best way was not to explicitly tell them that longer queries are better, but rather to present a halo around the search box that changes color as the query gets longer. A very interesting approach to apply persuasive technology to search!
• Then Rutgers student Roberto González-Ibáñez presented joint work with Chirag Shah and Ryen White on “Pseudo-Collaboration as a Method to Perform Selective Algorithmic Mediation in Collaborative IR Systems“. He presented a novel approach that identified when a user should be aided by a collaborator, and to what extent such help could enhance the user’s search success. An interesting way to achieve the benefits of both user-mediated and system-mediated collaboration.
• Finally, University of Washington student Jeff Huang presented joint work with Abdigani Diriye on “Web User Interaction Mining from Touch-Enabled Mobile Devices“. He focused on the practical concerns of instrumenting interaction with search engines in mobile environments. Specifically, he suggested tracking the viewport coordinates — that is, the visible portion of the page at any given time.

The short presentation format was extremely effective, encouraging presenters to communicate their ideas efficiently and leaving ample time for discussion.

Posters and Demos

As in previous years, we followed lunch with a vibrant session for posters and demos. Some of the more popular poster themes included question answering, task difficulty, and collaborative information seeking. Here is the full list of poster / demo presentations:

• Developing a Typology of Online Q&A Models and Recommending the Right Model for Each Question Type
  Erik Choi, Vanessa Kitzie, Chirag Shah
  
• Investigating Positive and Negative Affects in Collaborative Information Seeking: A Pilot Study Report
  Roberto González-Ibáñez, Chirag Shah
  
• To Ask or Not to Ask, That is The Question: Investigating Methods and Motivations for Online Q&A
  Vanessa Kitzie, Erik Choi, Chirag Shah
  
• Information Seeking Tasks: Why Do Searchers Feel Difficult?
  Jingjing Liu, Chang Suk Kim
  
• Finding Literary Themes with Relevance Feedback
  Aditi Muralidharan, Marti Hearst
  
• InFrame-Browsing: Enhancing Standard Web Search
  Marcus Nitsche, Andreas Nürnberger
  
• Trailblazer: Towards the Design of an Exploratory Search User Interface
  Marcus Nitsche, Andreas Nürnberger
  
• min: A Multi-Modal Web Interface for Math Search
  Christopher Sasarak, Kevin Hart, Siyu Zhu, Richard Pospesel, David Stalnaker, Lei Hu, Robert Livolsi, Richard Zanibbi
  
• Search Tactics in Collaborative Exploratory Web Search
  Zhen Yue, Shuguang Han, Daqing He
  
• Developing a Dual-Process Information-Seeking Model for Exploratory Search
  Michael Zarro
  
• Interactive Data Mining at the Speed of Thought
  Vladimir Zelevinsky
  
• Do Users with Different Domain Knowledge Select Different Sets of Documents?
  Xiangmin Zhang, Jingjing Liu, Xiaojun Yuan, Michael Cole, Nicholas Belkin, Chang Liu
  
• Predicting Task Difficulty from a User’s Moment to Moment Cognitive Effort During Information Seeking
  Michael Cole, Jacek Gwizdka, Chang Liu, Nicholas Belkin
  
• Effects of Domain Knowledge on User Task Performance in a Knowledge Domain Visualization System
  Xiaojun Yuan, Chaomei Chen, Xiangmin Zhang, Joshua Avery, Tao Xu
  
• Investigating the Effect of Visualization on User Performance of Information Systems
  Xiaojun Yuan

Full Paper Presentations

The full paper presentations were split into two sessions, the first held on the 4th and the second held on the 5th. Each presentation slot was 30 minutes. The full papers will be made available soon through the ACM Digital Library.

• University of Magdeburg student Marcus Nitsche presented “Knowledge Journey: A Web Search Interface for Young Users”, joint work with Tatiana Gossen and Andreas Nürnberger. The authors performed a study in which they found that children liked having personalized avatars that offer guidance, a wheel-shaped browsing menu, and a coverflow-style results presentation. It will be interesting to see how their study holds up in larger-scale user studies, and whether adults like some of these interface elements too.
• Oregon State University professor Carlos Jensen presented “Leyline: Provenance-Based Search Using a Graphical Sketchpad”, joint work with Seyedsoroush Ghorashi. I was intrigued to see a search approach focused entirely on provenance — that is, the history of a document’s ownership and transformations. I’m particularly curious about this area, since I’m a committee member for Aleatha Parker-Wood, who is pursuing a dissertation on “Making Sense of File Systems Through Provenance and Rich Metadata“.
• University of Waterloo professor Mark Smucker presented joint work with Charlie Clarke on “Modeling User Variance in Time-Biased Gain”. Their simulation-based approach produced distributions of gain that agree with distributions produced by real users. By emphasizing the effect size of differences, their approach could help uncover how much the performance differences among systems matter to real users.
• Finally, University of North Carolina at Chapel Hill professor Barbara Wildemuth and University of British Columbia professor Luanne Freund delivered a highly interactive presentation on “Assigning Search Tasks Designed to Elicit Exploratory Search Behaviors”. They performed an extensive survey of information exploration literature to identify concepts that authors have used to characterize exploratory search tasks. They tested examples on the audience to see how well we agreed with their characterization and with one another.

HCIR Challenge

With Friday morning came the most anticipated event of the Symposium: the HCIR Challenge. The Challenge is now in its third year: the 2010 Challenge focused on historical exploration of news using the New York Times Annotated Corpus; the 2011 Challenge focused on the problem of information availability using the CiteSeer digital library of scientific literature.

This year, we turned to the problem of people and expertise finding, a topic of obvious personal interest. We are grateful to Mendeley for providing this year’s corpus: a database of over a million researcher profiles with associated metadata including published papers, academic status, disciplines, awards, and more taken from Mendeley’s network of 1.6M+ researchers and 180M+ academic documents.

We asked participants to build systems that could perform three kinds of tasks:

1. Hiring. Given a job description, produce a set of suitable candidates for the position.
2. Assembling a Conference Program. Given a conference’s past history, produce a set of suitable candidates for keynotes, program committee members, etc. for the conference.
3. Finding People to deliver Patent Research or Expert Testimony. Given a patent, produce a set of suitable candidates who could deliver relevant research or expert testimony for use in a trial. These people can be further segmented, e.g., students and other practitioners might be good at the research, while more senior experts might be more credible in high-stakes litigation.

Each of the 5 teams was given 30 minutes to present.

• École Polytechnique Fédérale de Lausanne student Na Li presented “Magnifico: A Platform For Expert Mining Using Metadata“, joint work with Lei Zhou and Denis Gillet. Magnifico used a modified TF-IDF approach — where the IDF is an inverse discipline frequency — to match search queries to topic experts. It also assigned a multi-disciplinary reputation metric based on the expertise distribution of an author’s readers.
• Ben-Gurion University student Dima Kagan presented “Social Network Based Search for Experts“, joint work with Yehonatan Bitton, Michael Fire, Bracha Shapira, Lior Rokach, and Judit Bar-Ilan. Their system made excellent use of additional publicly available data, cross-referencing the Mendeley user profiles with data from Academia.edu and using Microsoft Academic Search to categorize publication and journals. You can try out their application here.
• University of Pittsburgh student Shuguang Han presented “IRIS-IPS: An Interactive People Search System for HCIR Challenge“– joint work with Daqing He, Zhen Yue, Jiepu Jiang, and Wei Jeng. The system used three different types of evidence to suggest candidates: expertise relevance, authority based on a PageRank algorithm applied to the co-authorship network, and social similarity using the Jaccard similarity between co-authors.
• Luanne Freund and Kristof Kessler, both from the University of British Columbia, presented “Exposing and exploring academic expertise with Virtu“, joint work with Michael Huggett and Edie Rasmussen. Virtu takes a task-based approach to expertise, exposing and giving the user control over dimensions of expertise that are more or less desirable depending on the type of expert-finding task. The search interface supports information interaction and exploration through a number of browsing and filtering tools, including facets and sliders. You can try out their application here.
• UCLA student Fei Liu presented the “‘iF’ People Search System“, an impressive solo effort. Also unique among the entries, iF is a mobile application, designed for the iPad and supporting swipe and multi-touch gestures. A very slick application, iF offered a novel approach to exploring the corpus of documents and people using the analysis of their reputations and social network relationships.

THE WINNER: Virtu! The competition was fierce, but Virtu stood out for the compelling approach it took to offering users control over the expert-finding process. Congratulations to Luanne, Kristof, and their colleagues for their outstanding work and well-deserved honor.

Reception

After we wrapped up the first day of the Symposium, we walked over to the nearby Technique, a restaurant in the Athenaeum Press building (home to two of Endeca’s offices in our early years) where students of Le Cordon Bleu practice their culinary skills. I’m no master chef, but I certain hope these students earned excellent grades for their performance. We enjoyed a delightful sampling of wines, appetizers, main courses, and desserts.

Conclusion

HCIR has been getting better every year, and this year was no exception. Many attendees in previous years had felt that the one-day format made the event feel rushed, and expanding to a second day took off much of the time pressure. We had ample opportunity for discussion, during the presentations as well as at the coffee breaks and reception. Finally, the Challenge was our best yet, eliciting extraordinary results from the five participating teams.

I’m proud of how far we’ve taken HCIR in these six years, and especially grateful to co-organizers Robert Capra, Gene Golovchinsky, Bill Kules, Catherine Smith, and Ryen White.

Time to start thinking about HCIR 2013!

I’ll be in Cambridge, MA this Thursday and Friday for the Sixth Symposium on Human-Computer Interaction and Information Retrieval (HCIR 2012). Hope to see many of you there!

But I’ll also have a few hours on Wednesday evening to meet people informally. If you’re interested in learning more about LinkedIn, data science or anything else, then hop over to the  Cambridge Brewing Company on Wednesday, October 3rd. I should be there by 5pm, assuming my flight arrives on time, and I’ll plan to stay there though dinner. I’m pretty easy to contact, so feel free to reach out to me through the usual channels.

MIT and Harvard students and faculty are especially welcome!

It’s the event you’ve been waiting for: the Sixth Symposium on Human-Computer Interaction and Information Retrieval! HCIR 2012 will take place October 4th and 5th at IBM Research in Cambridge, MA.

Who should attend?

Researchers, practitioners, and anyone else interested in the exciting work at the intersection of HCI and IR. Areas like interactive information retrieval, exploratory search, and information visualization.

Why attend?

You’ll enjoy a highly interactive day and a half of learning from HCIR leaders and pioneers. People like keynote speaker Marti Hearst, who literally wrote the book on search user interfaces. Folks from top universities and industry labs who are developing new methods and models for information seeking. And you’ll get to see the five teams competing to win the third annual HCIR Challenge, focused this year on people and expertise finding.

How to register?

Just click here and fill out the information requested. The $150 registration fee includes all sessions on both days, all meeting materials, a reception on October 4th at Technique. We’re grateful to our sponsors — FXPAL, IBM Research, LinkedIn, Microsoft Research, MIT CSAIL, and Oracle — for helping us keep the costs so low.

Capacity is limited, so please register as soon as possible to ensure your attendance.

Daniel Tunkelang: Content, Connections, and Context

Mario Rodriguez, Christian Posse, and Ethan Zhang: Multiple Objective Optimization in Recommender Systems

Anmol Bhasin: Beyond Ratings and Followers

Mohammad Amin, Baoshi Yan, Sripad Sriram, Anmol Bhasin, and Christian Posse: Social Referral: Leveraging Network Connections to Deliver Recommendations

I spent the past week in Dublin attending the 6th ACM International Conference on Recommender Systems (RecSys 2012). This young conference has become the premier global forum for discussing the state of the art in recommender systems, and I’m thrilled to have has the opportunity to participate.

Sunday: Workshops

The conference began on Sunday with a day of parallel workshops.

I attended the Workshop on Recommender Systems and the Social Web, where I presented a keynote entitled “Content, Connections, and Context“. Major worktop themes included folksonomies, trust, and pinning down what we mean by “social” and “context”. The most interesting presentation was “Online Dating Recommender Systems: The Split-complex Number Approach“, in which Jérôme Kunegis modeled the dating recommendation problem (specifically, the interaction of “like” and “is-similar” relationships) using a variation of quaternions introduced in the 19th century! The full workshop program, including slides of all the presentations is available here.

Unfortunately, I was not able to attend the other workshops that day, which focused on Human Decision Making in Recommender Systems, Context-Aware Recommender Systems (CARS), and Recommendation Utility Evaluation (RUE). But I did hear that Carlos Gomez-Uribe delivered an excellent keynote at the RUE workshop on the challenges of offline and online evaluation of Netflix’s recommender systems.

Monday: Experiments, Evaluations, and Pints All Around

Monday started with parallel tutorial sessions. I attended Bart Knijnenburg‘s tutorial on “Conducting User Experiments in Recommender Systems“. Bart is an outstanding lecturer, and he delivered an excellent overview of the evaluation landscape. My only complaint is that there was too much material for even a 90-minute session. Fortunately, his slides are online, and perhaps he’ll be persuaded to expand them into book form. Unfortunately, I missed Maria Augusta Nunes and Rong Hu‘s parallel tutorial on personality-based recommender systems.

Then came a rousing research keynote by Jure Leskovec on “How Users Evaluate Things and Each Other in Social Media“. I won’t try to summarize the keynote here — the slides of this and other presentations are available online. But the point Jure made that attracted the most interest was that voting is so predictable that results are determined mostly by turn-out. Aside from the immediate applications of this observation to the US presidential elections, there are many research and practical questions about how to obtain or incent a representative participant pool — a topic I’ve been passionate about for a long time.

The program continued with research presentations on multi-objective recommendation and social recommendations. I may be biased, but my favorite presentation was the work that my colleague Mario Rodriguez presented on multiple-objective optimization in LinkedIn’s recommendation systems. I’ll post the slides and paper here as soon as they are available.

Monday night, we went to the Guinness Storehouse for a tour that culminated with fresh pints of Guinness in the Gravity Bar overlooking the city. We’re all grateful to William Gosset, a chemist working for the Guinness brewery when he introduced the now ubiquitous t-test in 1908 as a way to monitor the quality of his product. A toast to statistics and to great beer!

Tuesday: Math, Posters, and Dancing

Tuesday started with another pair of parallel tutorial sessions. I attended Xavier Amatriain‘s tutorial on “Building Industrial-scale Real-world Recommender Systems” at Netflix. It was an excellent presentation, especially considering that Xavier had just come from a transatlantic flight! A major theme in his presentation was that Netflix is moving beyond the emphasis on user ratings to make the interaction with the user more transparent and conversational. Unfortunately I had to miss the parallel tutorial on the “The Challenge of Recommender Systems Challenges” by Alan Said, Domonkos Tikk, and Andreas Hotho.

Tuesday continued with research papers on implicit feedback and context-aware recommendations. One that drew particular interest was Daniel Kluver’s information-theoretical work to quantify the preference information contained in ratings and predictions, measured in preference bits per second (paper available here for ACM DL subscribers). And Gabor Takacs had the day’s best line with “if you don’t like math, leave the room.” He wasn’t kidding!

Then came the posters and demos — first a “slam” session where each author could make a 60-second pitch, and then two hours for everyone to interact with the authors while enjoying LinkedIn-sponsored drinks. There were lots of great posters, but my favorite was Michael Ekstrand‘s “When Recommenders Fail: Predicting Recommender Failure for Algorithm Selection and Combination“.

Tuesday night we had a delightful banquet capped by a performance of traditional Irish step dancing. The dancers, girls ranging from 4 to 18 years old, were extraordinary. I’m sorry I didn’t capture any of the performance on camera, and I’m hoping someone else did.

Wednesday: Industry Track and a Grand Finale

Wednesday morning we had the industry track. I’m biased as a co-organizer, but I heard resounding feedback that the industry track was the highlight of the conference. I was very impressed with the presentations by senior technologists at Facebook, Yahoo, StumbleUpon. LinkedIn, Microsoft, and Echo Nest. And Ronny Kohavi‘s keynote on “Online Controlled Experiments: Introduction, Learnings, and Humbling Statistics“ was a masterpiece. I encourage you to look at the slides for all of these excellent presentations.

Afterward came the last two research sessions, which included the best-paper awardee “CLiMF: Learning to Maximize Reciprocal Rank with Collaborative Less-is-More Filtering“. I’ve been a fan of “less is more” ever since seeing Harr Chen present a paper with that title at SIGIR 2006, and I’m delighted to see these concepts making their way to the RecSys community. In fact, I saw some other ideas, like learning to rank, crossing over from IR to RecSys, and I believe this cross-pollination benefits both fields. Finally, I really enjoyed the last research presentation of the conference, in which Smriti Bhagat talked about inferring and obfuscating user demographics based on ratings. The technical and ethical facets of inferring private data are topics close to my heart.

Finally, next year’s hosts exhorted this year’s participants to come to Hong Kong for RecSys 2013, and we heard the final conference presentation: Neal Lathia’s 100-euro-winning entry in the RecSys Limerick Challenge.

Thursday: Flying Home

Sadly I missed the last day conference-related activity: the doctoral symposium, the RecSys Data Challenge, and additional workshops. I’m looking forward to seeing discussion of these online, as well as reviewing the very active #recsys2012 tweet stream.

All in all, it was an excellent conference. LinkedIn, Netflix, and other industry participants comprised about a third of attendees, and there was a strong conversation bridging the gap between academic research and industry practice. I appreciate the focus of the nuances of evaluation, particularly the challenges of combining offline evaluation with online testing, and ensuring that the participant pool is robust. The one topic where I would have like to see more discussion was that of creating robust incentives for people to participate in recommender systems. Maybe next year in Hong Kong?

Oh, and we’re hiring!

This is keynote presentation I delivered at the Workshop on Recommender Systems and the Social Web, held as part of the 6th ACM International Conference on Recommender Systems (RecSys 2012):

Content, Connections, and Context 

Recommender systems for the social web combine three kinds of signals to relate the subject and object of recommendations: content, connections, and context.

Content comes first – we need to understand what we are recommending and to whom we are recommending it in order to decide whether the recommendation is relevant. Connections supply a social dimension, both as inputs to improve relevance and as social proof to explain the recommendations. Finally, context determines where and when a recommendation is appropriate.

I’ll talk about how we use these three kinds of signals in LinkedIn’s recommender systems, as well as the challenges we see in delivering social recommendations and measuring their relevance.

When I’m back from Dublin, I promise to blog about my impressions and reflections from the conference. In the mean time, I hope you enjoy the slides!

LinkedIn is an industry leader in the area of recommender systems – a place where big data meets clever algorithms and content meets social. If you’re one of the 175M+ people using LinkedIn, you’ve probably noticed some of our recommendation products, such People You May Know, Jobs You Might Be Interested In, and LinkedIn Today.

So it’s no surprise we’re participating in the 6th ACM International Conference on Recommender Systems (RecSys 2012), which will take place in Dublin next week.

Here’s a preview:

• Sunday, September 9: I’ll be presenting a keynote at the Workshop on Recommender Systems and the Social Web entitled “Content, Connections, and Context“. Anmol Bhasin will be a panelist at the Workshop on Context-Aware Recommender Systems.
• Monday, September 10: Mario Rodriguez will present work he’s done with Christian Posse and Ethan Zhang on “Multiple Objective Optimization in Recommendation Systems“.
• Tuesday, September 11: LinkedIn is sponsoring the poster/demo session. Check out the session and stop by our booth!
• Wednesday, September 12: Anmol Bhasin will deliver an invited talk on “Recommender Systems & The Social Web” as part of the industry track that Yehuda Koren and I are co-chairing.

I hope to see many of you at the conference, especially if you’re interested in learning about opportunities to work on recommendation systems and related areas at LinkedIn. And perhaps you can provide your own recommendations — specifically, local pubs where we can take in the local spirit.

See you in Dublin. Sláinte!

Sharing knowledge is part of our core culture at LinkedIn, whether it’s through hackdays or contributions to open-source projects. We actively participate in academic conferences, such as KDD, SIGIR, RecSys, and CIKM, as well as industry conferences like QCON and Strata.

Beyond sharing our own knowledge, we provide a platform for researchers and practitioners to share their insights with the technical community. We host an Tech Talk series at our Mountain View headquarters that we open up to the general public. Some of our recent speakers include Coursera founders Daphne Koller and Andrew Ng, UC-Berkeley professor Joe Hellerstein,  and Hadapt Chief Scientist Daniel Abadi. It’s an excellent opportunity for people with shared professional interests can reconnect with people they know, as well as make new connections. For those who cannot attend, we offer a live stream.

Our next talk will be by Panos Ipeirotis, a professor at NYU and one of the world’s top experts on crowdsourcing. Here is a full description:

Crowdsourcing: Achieving Data Quality with Imperfect Humans
Friday, September 7, 2012 at 3:00 PM
LinkedIn (map)

Crowdsourcing is a great tool to collect data and support machine learning — it is the ultimate form of outsourcing. But crowdsourcing introduces budget and quality challenges that must be addressed to realize its benefits.

In this talk, I will discuss the use of crowdsourcing for building robust machine learning models quickly and under budget constraints. I’ll operate under the realistic assumption that we are processing imperfect labels that reflect random and systematic error on the part of human workers. I will also describe our “beat the machine” system engages humans to improve a machine learning system by discovering cases where the machine fails and fails while confident on being correct. I’ll use classification problems that arise in online advertising.

Finally, I’ll discuss our latest results showing that mice and Mechanical Turk workers are not that different after all.

Panos Ipeirotis is an Associate Professor and George A. Kellner Faculty Fellow at the Department of Information, Operations, and Management Sciences at Leonard N. Stern School of Business of New York University. His recent research interests focus on crowdsourcing and on mining user-generated content on the Internet. He received his Ph.D. degree in Computer Science from Columbia University in 2004, with distinction. He has received three “Best Paper” awards (IEEE ICDE 2005, ACM SIGMOD 2006, WWW 2011), two “Best Paper Runner Up” awards (JCDL 2002, ACM KDD 2008), and is also a recipient of a CAREER award from the National Science Foundation.

If you’re in the Bay Area, I encourage you to attend in person — Panos is a great speaker, and it’s also a great opportunity to network with other attendees. If not, then you can follow on the live stream.

The event is free, but please sign up on the event page. See you next week!

Thank you Scott Adams for the free advertising. Of course, LinkedIn is the place to find data werewolves.

Want to find more data werewolves. Check out my team! Don’t worry, they only bite when they’re hungry.

Today we (specifically, my colleague Daria Sorokina) had the pleasure of hosting UT-Austin professor Matt Lease at LinkedIn to give a talk on his “Recent Adventures in Crowdsourcing and Human Computation“. It was a great talk, and the slides above are full of references to research that he and his colleagues have done in this area. A great resource for people interested in the theory and practice of crowdsourcing!

If you are interested in learning more about crowdsourcing, then sign up for an upcoming LinkedIn tech talk by NYU professor Panos Ipeirotis on “Crowdsourcing: Achieving Data Quality with Imperfect Humans“.

And if you’re already an expert, then perhaps you’d like to work on crowdsourcing at LinkedIn!

Using average values of traditional evaluation measures [for information retrieval systems] is not an appropriate methodology because it emphasizes effective topics: poorly performing topics’ scores are by definition small, and they are therefore difficult to distinguish from the noise inherent in retrieval evaluation.

Ellen is one of the world’s top researchers in the field of information retrieval evaluation. And for those not familiar with TREC terminology, “topics” are the queries used to evaluate information retrieval systems. So what she’s saying above is that, in order to evaluate systems effectively, we need to focus more on failures than on successes.

Specifically, she proposed that we judge information retrieval system performance by measuring the percentage of topics (i.e., queries) with no relevant results in the top 10 retrieved (%no), a measure that was then adopted by the TREC robust retrieval track.

Information Retrieval in the Wild

Information retrieval (aka search) in the wild is a bit different from information retrieval in the lab. We don’t have a gold standard of human relevance judgements against which we can compare search engine results. And even if we can assemble a representative collection of test queries, it isn’t economically plausible to assemble this gold standard for a large document corpus where each query can have thousands — even millions — of relevant results.

Moreover, the massive growth of the internet and the advent of social networks have changed the landscape of information retrieval. The idea that the relationship between a document and a search query, would  be sufficient to determine relevance was always a crude approximation, but now the diversity of a global user base makes this approximation even cruder.

For example, consider this query on Google for [nlp]:

Hopefully Google’s hundreds of ranking factors — and all of you —  know me well enough to know that, when I say NLP, I’m probably referring to natural language processing rather than neuro-linguistic programming. Still, it’s an understandable mistake — the latter NLP sells a lot more books.

And search in the context of a social network makes the user’s identity and task context key factors for determine relevance — factors that are uniquely available to each user. For example, if I search on Linkedin for [peter kim], the search engine cannot know for certain whether I’m looking for my former co-worker, a celebrity I’m connected to, a current co-worker who is a 2nd-degree connection, or someone else entirely.

In short, we cannot rely on human relevance judgments to determine if we are delivering users the most relevant results.

From %no to WTF! @ k

But human judgments can still provide enormous value for evaluating search engine and recommender system performance. Even if we can’t use them to distinguish the most relevant results, we can identify situations where we are delivering glaringly irrelevant results. Situations where the user’s natural reaction is “WTF!“.

People understand that search engines and recommender systems aren’t mind readers. We humans recognize that computers make mistakes, much as other people do. To err, after all, is human.

What we don’t forgive — especially from computers — are seemingly inexplicable mistakes that any reasonable person would be able to recognize.

I’m not going to single out any sites to provide examples. I’m sure you are familiar with the experience of a search engine or recommender system returning a result that makes you want to scream “WTF!”. I may even bear some responsibility, in which case I apologize. Besides, everyone is entitled to the occasional mistake.

But I’m hard-pressed to come up with a better measure to optimize (i.e., minimize) than WTF! @ k — that is, the number of top-k results that elicit a WTF! reaction. The value of k depends on the application. For a search engine, k = 10 could correspond to the first page of results. For a recommender system, k is probably smaller, e.g., 3.

Also the system can substantially mitigate the risk of WTF! results by providing explanations for results and making the information seeking process more of a conversation with the user.

Measuring WTF! @ k

Hopefully you agree that we should strive to minimize WTF! @ k. But, as Lord Kelvin tells us, if you can’t measure it, then you can’t improve it. How do we measure WTF! @ k?

On one hand, we cannot rely on click behavior to measure it implicitly. All non-clicks look the same, and we can’t tell which ones were WTF! results. In fact, egregiously irrelevant results may inspire clicks out of sheer curiosity. One of the phenomena that search engines watch out for is an unusually high click-through rate — those clicks often signal something other than relevance, like a racy or offensive result.

On the other hand, we can measure WTF! @ k with human judgments. A rater does not need to have the personal and task context of a user to evaluate whether a result is at least plausibly relevant. WTF! @ k is thus a measure that is amenable to crowdsourcing, a technique that both Google and Bing use to improve search quality. As does LinkedIn, and we are hiring a program manager for crowdsourcing.

Conclusion

As information retrieval systems become increasingly personalized and task-centric, I hope we will see more people using measures like WTF! @ k to evaluate their performance, as well as working to make results more explainable. After all, no one likes hurting their computer’s feelings by screaming WTF! at it.

But the first step is getting those people on your team. And hiring great people is hard, even when you and your colleagues are building the world’s best hiring solutions! By definition, the best people are scarce and highly sought after.

At the risk of giving away my competitive edge, I’d like to offer a word of advice to hiring managers: take it personally. That is, make the hiring process all about the people you’re trying to hire and the people on your team.

How does that work in practice? It means that everyone on the team participates in every part of the hiring process — from sourcing to interviewing to closing. A candidate interviews with the team he or she will work with, so everyone is invested in the process. The interview questions reflect the real problems the candidate would work on. And interviews communicate culture in both directions — by the end of the interviews, it’s clear to both the interviewers and the candidate whether they would enjoy working together.

I’ve seen and been part of impersonal hiring processes. And I  understand how the desire to build a scalable process can lead to a bureaucratic, assembly-line approach. But I wholeheartedly reject it. Hiring is fundamentally about people, and that means making the process a human one for everyone involved.

And taking it personally extends to sourcing. Earlier this week, the LinkedIn data science team hosted a happy hour for folks interested in learning more about us and our work. Of course we used our own technology to identify amazing candidates, but I emailed everyone personally, and the whole point of the event was to get to know one another in an informal atmosphere. It was a great time for everyone, and I can’t imagine a better way to convey the unique team culture we have built.

I’m all for technology and process that offers efficiency and scalability. But sometimes your most effective tool is your own humanity. When it comes to hiring, take it personally.

This year, I’m focusing my attention on three conferences: RecSys, HCIR, and CIKM. Regrettably I won’t be able to attend SIGIR, Strata NY, or UIST. But fortunately my colleagues are attending the first two, and hopefully some UIST attendees will be able to arrive a few days early and attend HCIR. Perhaps we can steal a page from WSDM and CSCW and arrange a cross-conference social in Cambridge.

6th ACM Recommender System Conference (RecSys 2012)

At RecSys, which will take place September 9-13 in Dublin,I’m co-organizing the Industry Track with Yehuda Koren. The program features technology leaders from Facebook, Microsoft, StumbleUpon, The Echo Nest, Yahoo, and of course LinkedIn. I’m also delivering a keynote at the Workshop on Recommender Systems and the Social Web. I hope to see you there, along with several of my colleagues who will be presenting their work on recommender systems at LinkedIn.

6th Symposium on Human-Computer Interaction and Information Retrieval (HCIR 2012)

The 6th HCIR represents a milestone — we’ve upgraded from a 1-day worksop to a 2-day symposium. We are continuing two great traditions: strong keynotes (Marti Hearst) and the HCIR Challenge (focused on people search). The symposium will take place October 4-5 in Cambridge, MA. Hope to see many of you there. And, if you’re still working on your submissions and challenge entries, good luck wrapping them up by the July 29 deadline!

21st ACM International Conference on Information and Knowledge Management (CIKM 2012)

Finally, you can’t miss CIKM in Hawaii! This year’s conference will take place October 29 – November 2 in Maui. After co-organizing last year’s industry track in Glasgow, I’m delighted to be a speaker in this year’s track, which also includes researchers and practitioners from Adobe, eBay, Google, Groupon, IBM, Microsoft, Tencent, Walmart Labs, and Yahoo. A great program in one of the world’s most beautiful settings, how can you resist?

I hope to see many of you at one — hopefully all! — of these great events! But, if you can’t make it, be reassured that I’ll blog about them here.

Our first HCIR Challenge in 2010 focused on exploratory search of a news archive. Thanks to the generosity of the Linguistic Data Consortium (LDC), we were able to provide participants with access to the New York Times (NYT) Annotated Corpus free of charge. Six teams presented their entries:

Search for Journalists: New York Times Challenge Report
Corrado Boscarino, Arjen P. de Vries, and Wouter Alink (Centrum Wiskunde and Informatica)

Exploring the New York Times Corpus with NewsClub
Christian Kohlschütter (Leibniz Universität Hannover)

Searching Through Time in the New York Times (WINNER)
Michael Matthews, Pancho Tolchinsky, Roi Blanco, Jordi Atserias, Peter Mika, and Hugo Zaragoza (Yahoo! Labs)
(covered in Technology Review: “A Search Service that Can Peer into the Future“)

News Sync: Three Reasons to Visualize News Better
V.G. Vinod Vydiswaran (University of Illinois), Jeroen van den Eijkhof (University of Washington), Raman Chandrasekar (Microsoft Research), Ann Paradiso (Microsoft Research), and Jim St. George (Microsoft Research) 

Custom Dimensions for Text Corpus Navigation
Vladimir Zelevinsky (Endeca Technologies)

A Retrieval System Based on Sentiment Analysis
Wei Zheng and Hui Fang (University of Delaware)

In 2011, we continued wth a Challenge focused on the problem of information availability. Four teams presented their systems to address this particularly difficult area of information retrieval:

FreeSearch – Literature Search in a Natural Way
Claudiu S. Firan, Wolfgang Nejdl, Mihai Georgescu (University of Hanover), and Xinyun Sun (DEKE Lab MOE, Renmin)

Session-based search with Querium (WINNER)
Gene Golovchinsky (FX Palo Alto Lab) and Abdigani Diriye (University College London)

GisterPro
David L.Ostby and Edmond Brian (Visual Purple)

Query Analytics Workbench
Antony Scerri, Matthew Corkum, Keith Gutfreund, Ron Daniel Jr., Michael Taylor (Elsevier Labs)

This year’s Challenge focuses on people search — that is, on the problem of people and expertise finding.

Here are examples of the kinds of tasks we will publish after the systems are frozen at the end of August:

• Hiring

  Given a job description, produce a set of suitable candidates for the position. An example of a job description: http://www.linkedin.com/jobs?viewJob=&jobId=3004979.
  
  

• Assembling a Conference Program

  Given a conference’s past history, produce a set of suitable candidates for keynotes, program committee members, etc. for the conference. An example conference could be HCIR 2013, where past conferences are described at http://hcir.info/.
  
  

• Finding People to deliver Patent Research or Expert Testimony

  Given a patent, produce a set of suitable candidates who could deliver relevant research or expert testimony for use in a trial. These people can be further segmented, e.g., students and other practitioners might be good at the research, while more senior experts might be more credible in high-stakes litigation. An example task would be to find people for http://www.articleonepartners.com/study/index/1658-system-and-method-for-providing-consumer-rewards.

For all of the tasks there is a dual goal of obtaining a set of candidates (ideally organized or ranked) and producing a repeatable and extensible search strategy.

Best of luck to this year’s HCIR Challenge participants — I’m excited to see the systems that they present this October at the Symposium!

I’m proud to be co-organizing the RecSys 2012 Industry Track with Yehuda Koren.

Check out the line-up:

• Ronny Kohavi (Microsoft), Keynote
  Online Controlled Experiments: Introduction, Learnings, and Humbling Statistics

• Ralf Herbrich (Facebook)
  Distributed, Real-Time Bayesian Learning in Online Services

• Ronny Lempel (Yahoo! Research)
  Recommendation Challenges in Web Media Settings

• Sumanth Kolar (StumbleUpon)
  Recommendations and Discovery at StumbleUpon

• Anmol Bhasin (LinkedIn)
  Recommender Systems & The Social Web

• Thore Graepel (Microsoft Research)
  Towards Personality-Based Personalization

• Paul Lamere (The Echo Nest)
  I’ve got 10 million songs in my pocket. Now what?

Hope to see you at RecSys this September in Dublin! Registration is open now.

A couple of years ago, I blogged about an information cascade problem I’d read about in David Easley and Jon Kleinberg‘s textbook on Networks, Crowds, and Markets. To recall the problem (which they themselves borrowed from Lisa Anderson and Charles Holt:

The experimenter puts an urn at the front of the room with three marbles in it; she announces that there is a 50% chance that the urn contains two red marbles and one blue marble, and a 50% chance that the urn contains two blue marbles and one red marble…one by one, each student comes to the front of the room and draws a marble from the urn; he looks at the color and then places it back in the urn without showing it to the rest of the class. The student then guesses whether the urn is majority-red or majority-blue and publicly announces this guess to the class.

The fascinating result is that the sequence of guesses locks in on a single color as soon as two consecutive students agree. For example, if the first two marbles drawn are blue, then all subsequent students will guess blue. If the urn is majority-red, then it turns out there is a 16/21 probability that the sequence will converge to red and a 5/21 probability that it will converge to blue.

Let me explain why I find this problem so fascinating.

Consider a scenario where you are among a group of people faced with the single binary decision — let’s say, choosing red or blue — and that each of you is independently tasked with recommending the best decision given your own judgement and all available information. Assume further that each of you is perfectly rational and that each of your prior decisions (i.e., without knowing what anyone else thinks) is based on independent and identically distributed random variables. Let’s follow the example above, in which each participant in the decision process has a prior corresponding to a Bernoulli random variable with probability p = 2/3.

If each of you makes a decision independently, then the expected fraction of participants who makes the right decision is 2/3.

But you could do better if you have a chance to observe others’ independent decision making first. For example, if you get to witness 100 independent decisions, then you have a very low probability of going wrong by voting the majority. If you’d like the gory details, review the cumulative distribution function of binomial random variables.

On the other hand, if the decisions happen sequentially and every person has access to all of the previous decisions, then we see an information cascade. Rationally, it makes sense to let previous decisions influence your own — and indeed 16/21 > 2/3. But 16/21 is still almost a one in four chance of making the wrong decision, even after you witness 100 previous decisions. We are wasting a lot of independent input because of how participants are incented.

I can’t help wondering how changing the incentives could affect the outcome of this process. What would happen if participants were rewarded based, in whole or in part, on the accuracy of the participants who guess after them?

Consider as an extreme case rewarding all participants based solely on the accuracy of the final participant’s guess. In that case, the optimal strategy for all but the last participant is to ignore previous participants’ guesses and vote based solely on their own independent judgements. Then the final participant combines these judgements with his or her own and votes based on the majority. The result makes optimal use of all participants’ independent judgments, despite the sequential decision process.

But what if individuals are reward based on a combination of individual and collective success? Consider the 3rd participant in our example who draws a red marble after the previous participants guess blue. Let’s say that there are 5 participants in total. If the reward is entirely based on individual success, the 3rd participant will vote blue, yielding an expected reward of 2/3. If the reward is entirely based on group success, the 3rd participant will vote red, yielding an expected reward of 20/27 (details left as an exercise for the reader). If we make the reward evenly split between individual success and group success, the 3rd participant will still vote blue — the benefit from helping the group will not be enough to overcome the cost to the individual reward.

There’s a lot more math in the details of this problem, e.g. “The Mathematics of Bayesian Learning Traps“, by Simon Loertscher and Andrew McLennan. But there’s a simple take-away: incentives are crucial in determining how we best exploit our collective wisdom. Something to think about the next time you’re on a committee.

This morning I had the pleasure to present a keynote address at the Semantic Technology & Business Conference (SemTechBiz). I’ve had a long and warm relationship with the semantic technology community — especially with Marco Neumann and the New York Semantic Web Meetup.

But I’m not exactly a fanboy of the semantic web, and I wasn’t sure how the audience would respond to some of my more provocative assertions. Fortunately the reception was very positive. Several people approached me afterwards to thank me for presenting a balanced argument for combining big data with structured representations and for raising HCIR issues.

A couple of people felt that faceted search was old news. I’m delighted that faceted search is becoming increasingly common, but there is still a lot of opportunity to use it more often and more effectively, And I was pleasantly surprised at the interest in discussing extensions of faceted search to address relationships between entities, as well as other nuances. I’ll have to dive into those in future posts.

For now, I hope you enjoy the slides, and I encourage you to ask questions in the comments.

Long-time readers know that I’m a great fan of visual search startup Modista, which was a victim of software patent abuse. To my delight, Modista is back from the dead. Check it out!

Also see my previous coverage of Modista.

The HCIR Symposium (formerly known as the HCIR Workshop) has run annually since 2007. The event unites academic researchers and industrial practitioners working at the intersection of HCI and IR to develop more sophisticated models, tools, and evaluation metrics to support activities such as interactive information retrieval and exploratory search. It provides an opportunity for attendees to informally share ideas via posters, small group discussions and selected short talks.

The Sixth Symposium on Human-Computer Interaction and Information Retrieval will be held as a two-day event on October 4 and 5, 2012 at IBM Research in Cambridge, Massachusetts. We are delighted to bring the event back to its birthplace (HCIR 2007 took place at MIT), and even more pleased to announce that our keynote speaker for the symposium this year will be UC Berkeley professor and search user interfaces pioneer Marti Hearst.

Topics for discussion and presentation at the symposium include, but are not limited to:

• Novel interaction techniques for information retrieval.
• Modeling and evaluation of interactive information retrieval.
• Exploratory search and information discovery.
• Information visualization and visual analytics.
• Applications of HCI techniques to information retrieval needs in specific domains.
• Ethnography and user studies relevant to information retrieval and access.
• Scale and efficiency considerations for interactive information retrieval systems.
• Relevance feedback and active learning approaches for information retrieval.

Demonstrations of systems and prototypes are particularly welcome.

We are also excited to continue the HCIR Challenge, this year focusing on the problem of people and expertise finding. We are grateful to Mendeley for providing this year’s corpus: a database based on Mendeley’s network of 1.6M+ researchers and 180M+ academic documents. Participants will build systems to enable efficient discovery of experts or expertise for applications such as collaborative research, team building, and competitive analysis.

In addition to the Challenge and a small number of research presentations, we will leave plenty of time for what participants have consistently told us that they find extremely valuable: informal discussions, posters and directed group discussions. Finally, we are extending our previous format to include a few full-length, fully-refereed archival quality papers that will be indexed in the ACM Digital Library.

We have extended the event to a second day to accommodate more presentations (including the full papers), and to leave plenty of time for discussion and for interaction around the poster session.  There will be a reception on Thursday evening.

Please consult the symposium web site,  http://www.hcir.info/hcir-2012, for full details. But here are some important dates to keep in mind:

• Deadline to request access to HCIR Challenge corpus: Friday, June 15
• Submission deadline for position and research papers: Sunday, July 29

This event would not be possible without generous support from industry and academia. This year’s supporters are IBM Research, LinkedIn, Microsoft Research, MIT CSAIL, and Oracle. Microsoft Research is also providing funds for a limited number of student travel awards. Information about these awards is available at  http://hcir.info/hcir-2012/student-travel.

Looking forward to seeing you in October!

Lots of people ask me what it’s like to be a data scientist at LinkedIn. The short answer: it’s awesome. Folks like Pete Skomoroch and team are building data products related to identity and reputation, such as Skills and InMaps. Yael Garten is leading the effort to understand and increase mobile engagement. And other folks work on everything from open-source infrastructure to fraud detection. Amazing people helping our 160M+ members by deriving valuable insights from big data.

I wanted to take a moment to showcase my own team. As a team, we straddle the boundary between science and engineering. We work closely with several engineering teams to deliver products that our members use everyday.

Joseph Adler is a name you might recognize from your bookshelf: he wrote Baseball Hacks and R in a Nutshell, both published by O’Reilly. At LinkedIn, he is a data hacker extraordinaire, currently focused on improving the network update stream.

Ahmet Bugdayci just joined LinkedIn this year, and he’s already on a tear. He’s working on a better approach to representing job titles, one of the most fundamental facets of our members’ professional identity. And he’s a polyglot.

Heyning Cheng is our innovator in chief. He envisions data products and does whatever it takes to hack them together. Our recruiters are especially happy to be his beta testers, and we’re working to turn those prototypes into shipped product.

Abhimanyu Lad is working on the next generation of LinkedIn search. He’s already improved spelling correction and group search, as well as building better ways to measure search effectiveness. But stay tuned — the best is yet to come!

Gloria Lau leads all things data for the student initiative. Check out LinkedIn Alumni to see what she’s been up to. Students are the future, and we’re excited to be making LinkedIn a great tools for students, alumni, and universities.

Monica Rogati spearheaded many of LinkedIn’s key products: the Talent Match system that matches jobs to candidates; the first machine learning model for People You May Know; and the first version of Groups You May Like. When she’s not working on our products, she gives awesome presentations.

Daria Sorokina recently joined us and is working on search quality. She’s a hard-core machine learning researcher and developer: check out her open-source code for additive groves.

Ramesh Subramonian has been focused on data efforts for our international expansion. Over 60% of our members live outside the United States, and his efforts ensure that LinkedIn’s value proposition is a global one.

Joyce Wang is a data science generalist. She is part of the search team, but she’s built great tools for log analysis and human evaluation that are finding great use across the company.

I hope that gives you a flavor of what it’s like to be a data scientist at LinkedIn — and on my team in particular.

Do you possess that rare combination of computer science background, technical skill, creative problem-solving ability, and product sense? If so, then I’d love to talk with you about opportunities to work on challenging problems with amazing people!

Last night, I had the pleasure to deliver the keynote address at the CIO Summit US. It was an honor to address an assembly of CIOs, CTOs, and technology executives from the nation’s top organizations. My theme was “Science as a Strategy”.

To set the stage, I told the story of TunkRank: how, back in 2009, I proposed a Twitter influence measure based on an explicit model of attention scarcity which proved better than the intuitive but flawed approach of counting followers. The point of the story was not self-promotion, but rather to introduce my core message:

Science is the difference between instinct and strategy.

Given the audience, I didn’t expect this message to be particularly controversial. But we all know that belief is not the same as action, and science is not always popular in the C-Suite. Thus, I offered three suggestions to overcome the HIPPO (Highest Paid Person’s Opinion):

• Ask the right questions.
• Practice good data hygiene.
• Don’t argue when you can experiment!

Asking the Right Questions

Asking the right questions seems obvious — after all, our answers can only be as good as the questions we ask. But science is littered with examples of people asking the wrong questions — from 19th-century phrenologists measuring the sizes of people’s skulls to evaluate intelligence to IT executives measuring lines of code to evaluate programmer productivity. It’s easy for us (today) to recognize these approaches as pseudoscience, but we have to make sure we ask the right questions in our own organizations.

As an example, I turned to the challenge of improving the hiring process. One approach I’ve seen tried at both Google and LinkedIn is to measure the accuracy of interviewers — that is, to see how well the hire / no-hire recommendations of individual interviewers predict the final decisions. But this turns out to be the wrong question — in large part because negative recommendations (especially early ones) weigh much more heavily in the decision than positive ones.

What we found instead was that we should focus on efficiency as an optimization problem. More specifically, there is a trade-off: short-circuiting the process as early as possible (e.g., after the candidate performs poorly on the first phone screen) reduces the average time per candidate, but it also reduces the number of good candidates who make it through the process. To optimize overall throughput (while keeping our high bar), we’ve had to calibrate the upstream filters. How to optimize that upstream filter turns out to be the right question to ask — and one we still continue to iterate on.

More generally, I talked about how, when we hire data scientists at LinkedIn, we look for not only strong analytical skills but also the product and business sense to pick the right questions to ask – questions whose answers create value for users and drive key business decisions. Asking the right questions is the foundation of good science.

Practicing Good Data Hygiene

Data mining is amazing, but we have to watch out for its pejorative meaning of discovering spurious patterns. I used the Super Bowl Indicator as an example of data mining gone wrong — with 80% accuracy, the division (AFC vs. NFC) of the Super Bowl champion predicts the coming year’s stock market performance. Indeed, the NFC won this year (go Giants!) and subsequent market gains have been consistent with this indicator (so far).

We can all laugh at these misguided investors, but we make these mistakes all the time. Despite what researchers have called the “unreasonable effectiveness of data”, we still need the scientific method of first hypothesizing and then experimenting in order to obtain valid and useful conclusions. Without data hygiene, our desires, preconceptions, and other human frailties infect our rational analysis.

A very different example is using click-through data to measure the effectiveness of relevance ranking. This approach isn’t completely wrong, but it suffers from several flaws. And the fundamental flaw relates to data hygiene: how we present information to users infects their perception of relevance. Users assume that top-­ranked results are more relevant than lower-­ranked results. Also, they can only click on the results presented to them. To paraphrase Donald Rumsfeld: they don’t know what they don’t know. If we aren’t careful, a click-­based evaluation of relevance creates positive feedback and only reinforces our initial assumptions – which certainly isn’t the point of evaluation!

Fortunately, there are ways to avoid these biases. We can pay people to rate results presented to them in random order. We can use the explore / exploit technique to hedge against the ranking algorithm’s preconceived bias. And so on.

But the key take-away is that we have to practice good data hygiene, splitting our projects into the two distinct activities of hypothesis generation (i.e., exploratory analysis) and hypothesis testing using withheld data.

Don’t Argue when you can Experiment

I couldn’t resist the opportunity to cite Nobel laureate Daniel Kahneman‘s seminal work on understanding human irrationality. I also threw in Mercier and Sperber’s recent work on reasoning as argumentative. The summary: don’t trust anyone’s theories, not even mine!

Then what can you trust? The results of a well-­‐run experiment. Rather than debating data-­‐free assertions, subject your hypotheses to the ultimate test: controlled experiments. Not every hypothesis can be tested using a controlled experiment, but most can be.

I recounted the story of how Greg Linden persuaded his colleagues at Amazon to implement shopping-cart recommendations through A/B testing, despite objections from a marketing SVP. Indeed, his work — and Amazon’s generally — has strongly advanced the practice of A/B testing in online settings.

Of course, A/B testing is fundamental to all of our work at LinkedIn. Every feature we release, whether it’s the new People You May Know interface or improvements to Group Search relevance, starts with an A/B test. And sometimes A/B testing causes us to not launch — we listen to the data.

Don’t argue when you can experiment. Decisions about how to improve products and processes should not be by an Oxford-­style debate. Rather, those decisions should be informed by data.

Conclusion: Even Steve Jobs Made Mistakes

Some of you may think that this is all good advice, but that science is no match for an inspired leader. Indeed, some pundits have seen Apple’s success relative to Google as an indictment of data-­driven decision making in favor of an approach that follows a leader’s gut instinct. Are they right? Should we throw out all of our data and follow our CEOs’ instincts?

Let’s go back a decade. In 2002, Apple faced a pivotal decision – perhaps the most important decision in its history. The iPod was clearly a breakthrough product, but it was only compatible with the Mac. Remember that, back in 2002, Apple had only a 3.5% market share in the PC business. Apple’s top executives did their analysis and predicted that they could drive the massive success of the iPod by making it compatible with Windows, the dominant operating system with over 95% market share.

Steve Jobs resisted. At one point he said that Windows users would get to use the iPod “over [his] dead body”. After continued convincing, Jobs gave up. According to authorized biographer Walter Isaacson, Steve’s exact words were: “Screw it. I’m sick of listening to you assholes. Go do whatever the hell you want.” Luckily for Steve, Apple, and the consumer public, they did, and the rest is history.

It isn’t easy being one those ass­holes. But that’s our job, much as it was theirs. It’s up to us to turn data into gold, to apply science and technology to create value for our organizations. Because without data, we are gambling on our leaders’ gut feelings. And our leaders, however inspired, have fallible instincts.

Science is the difference between instinct and strategy.

Recently I had a couple of great opportunities to share my thoughts publicly, and I wanted to make sure readers here were aware of them.

The first was a special guest appearance on The Semantic Link, a program hosted by Paul Miller with regular panelists Peter Brown, Christine Connors, Eric Franzon, Eric Hoffer, Bernadette Hyland, and Andraz Tori. It was a lot of fun, and a great warm-up for the keynote I’ll be delivering on “Scale, Structure, and Semantics” at the upcoming Semantic Tech & Business Conference (SemTechBiz), which will take place in San Francisco in June.

The second was a live interview on Internet Evolution, hosted by Mary Jander and Nicole Ferraro. They clearly did their homework, scouring my blog posts and web commentary for everything controversial I’d ever said — and then some! If that’s enough to pique your interest, then I encourage you to listen to the recorded interview and read the chat transcript at Internet Evolution.

Happy to answer questions based on either of these sessions — comment away!

Note: This post was written by Yael Garten, a Senior Data Scientist at LinkedIn. Yael joined Linkedin in 2011, where she leads our mobile analytics team. She previously worked at Stanford on text mining, personalized medicine, and biomedical informatics.

We live in an era of Big Data. But how do we use all of that data to answer questions and communicate those answers effectively?

My colleagues and I at LinkedIn were fortunate enough to hear answers from Noah Iliinsky, who literally wrote the book on designing data visualization.

Earlier this month, we hosted Noah at LinkedIn to give a tech talk on “Designing Effective Data Visualizations“. We are proud to make these tech talks open to the public, and enjoyed a great mix of attendees from local companies and universities. If you couldn’t attend the talk in person or remotely, I encourage you to watch the recording, embedded above.

Why do we visualize data? As Noah tells us, visualization makes data accessible. It gives us faster access to actionable insights and allows access to huge amounts of data. Visualization enables both data exploration (when you are still trying to discover the story) and data explanation (when you have a story to tell). Noah reviewed some great examples (watch the talk!), with an emphasis on the dos and don’ts of data visualization.

In particular, he provided a step-by-step framework for traversing the path from question to answer:

Phase 1: Decide what to visualize.

• Understand the question your audience wants to answer.
• Understand the actions they are hoping the answer will drive.
• Consider who is consuming this data — their needs, biases, etc.
• Decide what data to use — and what data not to use — and what relationships you are interested in.
• Explore the data and construct a storyline.

Phase 2: Decide how to visualize it.

• Use appropriate visual encodings for data and relationships (cf. http://complexdiagrams.com/properties).
• Limit the data you include.
• Use position for your most important relationship.
• Try different axes.
• Show your visualization to different people, without explanations. Show an expert, show a layman.
• Iterate, iterate, iterate!

Noah also shared his thoughts on how to visualize social networks. He recommended useful tools for data visualization, including Tableau, Spotfire, D3, Processing, ggplot2, Omnigraffle, and OmnigraphSketcher.

• You are not your audience. This is a huge lesson that all of us must internalize to be great at what we do. Consider what you need to communicate to marketers, investors, member of the general public, etc.
• Do user research! Understand your users’ hopes, dreams, and favorite flavors! Understand their identity, their jargon, culture, etc.
• Remember that your success is defined by your customer’s success. If you can’t satisfy your customer’s needs, you have failed — no matter how insightful your analysis.

You can enjoy the talk by watching the embedded video above. And you can find more LinkedIn tech talks on our YouTube channel.

One of the highlights of the recent Data 2.0 Summit was a panel featuring:

• Alexander Gray, CTO of SkyTree
• Anthony Goldbloom, CEO of Kaggle
• Josh Wills, Director of Data Science at Cloudera

The focus of the panel was supposed to be about “Data Science and Predicting the Future”, but the most contentious topic was whether data, algorithms or people (that is, the data scientists themselves) were the most important factor in the practice and success of data science.

Yes, we one-upped the debate that my colleague Monica Rogati instigated at this year’s Strata conference. In fact, Josh cited the “better data beats more data beats clever algorithms” argument that Monica made in her own Strata presentation. And, just like at Strata, there was a healthy dose of audience participation.

Of course, I came down on the side of data — which I believe won the debate hands down.

I’m a fan of clever algorithms, which Alexander had to defend given that Skytree’s core value proposition is better machine learning algorithms delivered at scale. But I’m with Peter Norvig et al. on the dominance of data over algorithms.

Favoring data over people was a harder choice. Anthony naturally made the case for people (Kaggle’s claim to fame is assembling many of the world’s best data scientists by organizing competitions). Hopefully my team won’t quit en masse when they read this blog post! But I think they’ll agree with me that, without the incredible data we work with at LinkedIn, they’d be unable to deliver the awesomeness that I’ve come to expect from them.

There’s a saying that we all cook from the same cookbooks, so that it’s the ingredients that make all the difference. To take the metaphor further, you can also try to poach your rival’s chefs. But data is the biggest entry barrier — and the most sustainable competitive advantage.

Of course, we should have the best people apply the best algorithms to work with the best data. But data comes first. The best meal starts with the best ingredients.

The video of the presentation that Claire Hunsaker and I delivered on “Humans, Machines, and the Dimensions of Microwork” at Strata 2012 is now available as part of the complete video compilation. I’ve taken the liberty to upload it to YouTube — feel free to watch the embedded video above.

I’ll be participating in the Data 2.0 Summit on Tuesday, April 3rd, and I hope to see some of you there. Last year, my colleague (and fellow LinkedIn data scientist) Scott Nicholson attended and wrote this guest post about it. This year, I’m not only attending but participating on a panel about social data, moderated by AllthingsD Senior Editor Liz Gannes.

There’s a great line-up of speakers for the day, including:

• Bram Cohen, the founder, chief scientist, and inventor of BitTorrent, the leading peer-to-peer file sharing protocol for sharing large files on the Internet.
• Michael Driscoll, CTO and co-founder of the Metamarkets Group. He moderated a fantastic debate at the recent Strata conference about the relative importance of domain expertise or machine learning for data scientists.
• Gil Elbaz, the founder and CEO of Factual, an information marketplace. He is also the co-founder of Applied Semantics, which Google acquired in 2003 for $102M and turned into the foundation for AdSense (now a $10B business).
• Anthony Goldbloom, co-founder and CEO of Kaggle,  a platform for data science competitions that generated a lot of discussion at Strata.
• Stefan Weitz, director of search at Bing. He’ll be on my panel. Also see the discussion I had with him in the comment thread for a post on “Why Are People So Clueless About Search?“.

And lots more, but you get the idea. I’m thrilled to be part of such a talent-heavy program and looking forward to insightful discussions with with fellow panelists and attendees. Also a great excuse to spend a day in the city (note for my former townspeople — that’s what they call San Francisco around here).

Conventional wisdom holds that physical compliments are counter-productive as pick-up lines. Indeed, a dating site did some analysis showing a negative correlation between such compliments and the probability of a positive response.

But, as m6d Chief Scientist and 3-time KDD Cup winner Claudia Perlich explained in her recent talk at LinkedIn, we have to watch out for confounding variables. In the dating scenario above, beauty is a confounding variable: it determines both the probability of getting a positive response and of the probability of a suitor offering physical compliments. Hence, we need to control for the actual beauty or it can appear that making compliments is a bad idea.

Perlich does not work on online dating, but rather in the data-driven world of online advertising. Specifically, she and her team work on real-time bidding optimization.

Perlich described a variety of design choices that have general applicability to data science problems. For example, her team used hashed tokens of previously visited URLs, rather than the URLs themselves, as features for their machine learning models. They avoided the use of personally identifying information (PII) or even demographic information about their users. These decisions were counterintuitive — typically, more data leads to better results. But Perlich found that these restrictions did not sacrifice accuracy, and had the further benefit of keeping their approach general rather than application- or customer-specific.

Perlich also described several technical challenges that her team had to overcome. For example, they found they could not sample users, so they instead sampled events — that is, visits, impressions, and conversions. They also found that their linear models tended to suffer from overfitting in their top predictions — a problem they resolved by introducing a spline model.

The talk was deeply technical and yet very relevant and accessible to a broad audience of data scientists and engineers. There’s much more content than fits in this small summary, so I encourage you to watch the video! And you can watch more LinkedIn tech talks here.

In the past few years, prosopagnosia, also known as “face blindness”, has received a fair amount of attention from researchers, as well as from the popular press.

My first exposure to the topic was Joshua Davis’s article entitled “Face Blind“, which appeared in Wired in November 2006. I was intrigued, especially since I’d long recognized that I had difficulty recognizing people by face. Perhaps the person who has done most to raise awareness of prosopagnosia is neurologist Oliver Sacks, who has prosopagnosia himself.

The Wired article inspired me to explore the subject. I discovered faceblind.org and found quizzes that tested for prosopagnosia. On one of these, where random guessing would have earned a score of 50%, I scored in the low 60s. My initial reaction was that my score wasn’t so bad — it was a hard test! Then my wife took the test and scored in the high 90s. That’s when I realized that I didn’t just have difficulty recognizing faces — I was almost incapable of it.

Faced with this realization, I had to decide whether to share it with my friends and family, let alone with my broader set of social and professional acquaintances. It was tempting not to — after all, why tell the world that I wasn’t “normal”?

But eventually I realized that it would be better for people around me to know than not know. The biggest downside to prosopagnosia isn’t the momentary embarrassment of not recognizing someone — it’s the content fear of offending people who may think you don’t value them enough to recognize or acknowledge them.

Hence, I spread the word through my colleagues, ensuring that most of the people with whom I interacted regularly would find out without any big announcements. Some of my co-workers were surprised, since I do a pretty good job of recognizing people using non-facial clues — height, hair, clothing, where I run into them, etc. I have a great memory, and I have no problems with voice recognition. In other words, I have lots of work-arounds.

Fortunately, I work with a lot of people who understand machine learning — which is a great framework for understanding how I recognize people. I simply work with a different set of features than most people, but fortunately I achieve sufficient precision and recall to pass as “normal” most of the time.

Anyway, if you didn’t already know that I had prosopagnosia, welcome to the inner circle! And if you ever felt that I walked by you without recognizing or acknowledging you, please accept my belated apology.

Finally, if you’re curious to learn more about prosopagnosia, I encourage you to watch the 60 Minutes segments above.

Just watch the presentation, which stole the show at Strata 2012. The written word cannot do justice to Avinash’s passion and his extraordinary ability to communicate it.

I spent the last three days at the O’Reilly Strata Conference, an assembly of two thousand over 2500 people focused on data science and its applications. While I’m wary of industry conferences from attending vendor-fests in my past life in the enterprise software world, Strata is an exceptionally good conference. The speakers were a who’s who of data science, including Lucene and Hadoop creator Doug Cutting, search user interface pioneer Marti Hearst, and Google chief economist Hal Varian. You can find the tweet stream for the conference at hash tag #strataconf.

Tuesday

I spent Tuesday in the Deep Data session, billed as a no-holds-barred program for data scientists. My two favorite talks:

• Claudia Perlich, winner of three KDD cups, talked about using information to pick the right action and to influence people such that they behave in a way that is better for them, better for us, and possibly better for society in general.
• Monica Rogati, my colleague at LinkedIn and the epitome of a data scientist, delivered a fantastic talk about machine learning models and training data in the real world, extending Peter Norvig‘s point about the “unreasonable effectiveness of data” to observe that more data beats clever algorithms but better data beats more data.

But the most fun that day was the Oxford-style debate featuring Drew Conway, Pete Skomoroch, Mike Driscoll, DJ Patil, Amy Heineike, Pete Warden, and Toby Segaran. The question proposed was absurdly Manichean: if you had to hire your first data scientist and could only hire one, would you pick a domain expert or a machine learning expert? After the moderator suppressed some initial attempts to hedge (“both”, “it depends”, etc.), the debaters ripped into the question by taking extreme positions and defending them with gusto. It was a lot of fun, with enthusiastic audience participation and the debaters exploiting their inside knowledge of their opponents’ work histories. In the end, the machine learning side won by a small margin.

I then had the good fortune to grab dinner with Marti Hearst and Hal Varian at Xanh – a wonderful mix of great food and conversation.

Wednesday

The Wednesday morning keynote session offered some gems:

• Cloudera CEO Mike Olson urged big data practitioners to focus on guns, drugs, and oil.
• Doctor and data geek Ben Goldacre delivered a mesmerizing and disturbing talk about the suppression of inconvenient medical trial results and analytical tools to discover it.

But the person who stole the show was Google’s Avinash Kaushik, who talked about making love with data to find orgasm-inducing actions to change the world and make more money. Unfortunately this was the one talk that was not recorded, but you can read the summary on Avinash’s Google+ page.

As a speaker, I held “office hours” on Wednesday. It was supposed to be a 40-minute slot for conference attendees to come and ask me question. But somehow those 40 minutes extended into three hours of conversation about everything from normalized KL divergence to interview problems — and segued into a reception with specialty big-data cocktails. By the time I got back to my apartment, my voice, brain, and liver were spent.

Thursday

I spent most of Thursday morning in the speaker lounge, recovering from the previous evening and making last touches on my presentation. But I couldn’t resist attending a two-part session on privacy. Indeed, this session was distinctive enough to merits it’s own hash tag: #strataprivacy.

The first part featured O’Reilly’s Alex Howard moderating Intelius Chief Privacy Officer Jim Adler and NYU PhD student Solon Barocas on a panel provocatively titled  ”If Data Wants to Be Free, is Privacy a Prison?” It was a great discussion, and I enjoyed the opportunity to offer my own provocative question through Twitter. Since the panelists were arguing that it was unethical to infer private facts from public data, I asked if they were trying to establish a new form of thoughtcrime.

The second panel, entitled “Pretty Simple Data Privacy“, featured Kaitlin Thaney from Digital Science, Betsy Masiello from Google, and John Wilbanks from the Kauffman Foundation for Entrepreneurship. Given that today was the first day of Google’s new privacy policy, there was no avoiding focus on the associated controversy. I did try to get Betsy to address my charge that Google doesn’t think users own their search history (cf. “Google vs. Bing: A Tweetle Beetle Battle Muddle“), but she said she was unfamiliar with the details of that event. I do wish that someone at Google with more familiarity would respond publicly.

Back to the speaker room after lunch, until my own talk with Samasource’s Claire Hunsaker on “Humans, Machines, and the Dimensions of Microwork“. I’ll post the slides (and there will be a video on the conference site), but the sound bite is that you need to keep crowdsourcing tasks simple, manage the trade-off between task value and difficulty, and watch out for systematic bias.

I wrapped up the conference by hearing William Gunn talk about how Mendeley is disrupting bibliometrics and perhaps the entire academic publishing and reputation ecosystem. I laud his ambition and wish him and Mendeley luck in this quest.

In summary, three days of great talks, conversations, and general enjoyment. My thanks to Strata organizers Edd Dumbill and Alistair Croll for putting together such an outstanding event and for giving me the opportunity to participate.

My excursion to Seattle was delightful, and I thought I’d share some details with readers.

I spent most of Friday at the University of Washington, meeting with graduating PhD students.  I’ve always known that UW is a top school, but I was particularly impressed with this batch. I was pleasantly surprised to see folks like Nodira Khoussainova and Kayur Patel working to bring together the often disparate worlds of databases, machine learning, and HCI in order to make people more effective at solving “big data” problems. I realize that I’m aiding and abetting other employers with whom I compete for top talent, but it would be wrong not to encourage everyone to find worthy challenges for these budding scientists.

I then went to the Space Needle to meet up with the WSDM 2012 crowd. Jaime Teevan and Eytan Adar outdid themselves, providing a great setting for folks to mingle, imbibe, and enjoy a spectacular view of Seattle.

Saturday I attended the “social” day of the WSDM conference.

Andrew Tomkins chaired the first morning session, which included Hila Becker‘s latest work on identifying event content in social media and Georgios Zervas presenting the work on the analyzing reputational effects of Groupon that triggered quite a controversy last September. After the break came the spotlight section — a great sequence of 5-minute presentations that in which researchers both summarized their contributions and lured attendees to visit their posters. I hope that more conferences adopt this format, which optimizes for communicating ideas and discourages long-winded expositions.

I then had the pleasure to have lunch with Jan Pedersen and friends at Blueacre Seafood — great food and even better conversation. We both noted the irony that, even though we are practically neighbors, we only seems to meet up at events like these..

I made it back to the conference in time to hear the two best-paper awardees: Adam Sadilek on “Finding Your Friends and Following Them to Where You Are” and Yaron Singer on “How to Win Friends and Influence People, Truthfully: Influence Maximization Mechanisms for Social Networks“. I highly recommend both papers, especially if you are interested in either social network prediction or the underlying economics of influence.

Another coffee break, and then the keynote: Hilary Mason on “The Secret Life of Social Links”. Hilary is a great speaker — I first met her when I invited her to the Workshop on Search and Social Media (SSM 2010) at WSDM 2010. She didn’t disappoint, and it’s great to see practitioners like her crossing the aisle to engage the academic community. Not to mention infusing their slides with lolcats.

The conference wrapped up at 5pm, but then we bussed over to Microsoft Research for the Social Search Social. That was a fun event designed to cross-pollinate the WSDM and CSCW communities. Meredith Ringel Morris, Gene Golovchinksy, Jeremy Pickens, Madhu Reddy, Chirag Shah, and Michael Twidale put together a great program of 45-second madness presentations and “speed-dating” to pair up WSDM and CSCW attendees. It was far too short, but a lot of fun. And some of us kept up the social spirit by grabbing dinner afterward at Blue C Sushi.

To everyone I met in the last couple of days: thanks for the great company and conversation! Keep sharing ideas and making data and science social.

First, I wanted to give readers a heads up that I’ll be in Seattle this Friday and Saturday. I’ll spend Friday afternoon at the University of Washington, meeting with some of their outstanding computer science doctoral students. My schedule filled up with unexpected haste! But if you’re on campus and urgently want to meet, let me know and I’ll see what I can do.

Saturday I’ll be attending the social track of WSDM 2012, the premier international ACM conference covering research in the areas of search and data mining on the Web. I’m excited about the program, as well as the opportunity to catch up with friends and make new ones. Back in 2010, I had the pleasure of co-organizing the Workshop on Search and Social Media (SSM 2010) and being the official ACM blogger for WSDM 2010. You can read my posts here.

Then, on Saturday evening, I’ll be heading to Microsoft Research to attend the Social Search Social (SSS 2012). Hats off to organizers Meredith Ringel Morris, Gene Golovchinksy, Jeremy Pickens, Madhu Reddy, Chirag Shah, and Michael Twidale for creating what looks to be a fun (and very social!) event. I’m especially looking forward to the 45-second “madness” presentations (in which I’m participating) and the “speed dating” to help cross-pollinate  the WSDM and CSCW communities.

Hope to see some of you there, and of course will share what I learn here at The Noisy Channel. I also encourage you to follow the tweet streams for #wsdm2012 and #sss2012.

As regular readers know, I have a deep affection for Carnegie Mellon University, where I did my graduate work. I’m happy to announce that two of my colleagues (both fellow CMU PhDs) will be giving talks at CMU in a couple of weeks, and I hope that some of you will have the opportunities to attend.

On Tuesday, February 7th, Abhimanyu Lad will be hosting an information session at 6pm in Scaife Hall, Room 214. Abhi is rock star on our data science team, and he’s been working on the next generation of LinkedIn search. You can get a taste of his work from his recent HCIR 2011 presentation, “Is it Time to Abandon Abandonment?“. Abhi will talk about a variety of technical challenges that data scientists and engineers are working on at LinkedIn.

On Thursday, February 9th, Paul Ogilvie will talk about “Where Big Data Meets Real-Time: Efficiently Indexing and Ranking News using Activity” at 3:30pm in GHC 6115. Paul is responsible for article relevance infrastructure and algorithms on LinkedIn Today, a great example of social navigation – not to mention a great success for users. Paul will talk about the technical details that make LinkedIn Today possible, including a novel use of inverted lists to efficiently index and support real-time updates to document representations.

And, even if you can’t make it to the talks, I encourage you to visit the LinkedIn booth at the EOC fair on Wednesday, February 8th. We’re looking for great software engineers and data scientists, and we’re especially interested in interns.

This is the season of annual reviews, at least at LinkedIn. Performance reviews can be daunting for both employees and managers — at least everywhere that I’ve worked. Not only are we as human beings terrible at delivering feedback, but we also receive bad advice as managers.

For example, many of us have learned the “feedback sandwich” method, a technique that doesn’t hold up to scientific validation. Watch the video below to see what Stanford professor Clifford Nass has learned from his experiments (see my review of his book here).

Here is what I suggest as a format for performance feedback, whether for writing your own self-assessment or delivering feedback to reports or peers on their performance:

1) What is your day job?

Everyone needs a day job — a mission with a crisp set of responsibilities and deliverables. If you don’t know what you’re responsible for delivering, you can’t assess how well you are delivering it. You should know and articulate your top priorities — at most three, with a clear #1. For further reading, I suggest the Quora discussion on OKRs (Objectives and Key Results), an idea pioneered by Intel and now used at top technology companies (including LinkedIn and Google).

2) How are you performing in your day job?

Hopefully you make more contributions than you can count. But make sure that your day job comes first. If you find that a disproportionate fraction of your contribution is outside your day job, then consider changing your day job. Your top priority is to meet (hopefully exceed!) the expectations for your day job — expectations you should set early and revisit regularly. Performance reviews are a great opportunity to brag.

3) What do you do beyond your day job?

Your day job should be strongly aligned with your team and company’s top priorities. But great employees contribute beyond their day job towards other team and company priorities. For example, talent is our top priority at LinkedIn, so we particularly value contributions to hiring and growing our talent. And, at least in every environment I’ve experienced, the best employees are those who help make others successful.

4) How do you want to grow?

This is really a two-part question. First, what do you want to do next? That could mean getting better at your day job, evolving your current responsibilities, or taking on a different role. Second, what are you doing to get there? You are ultimately responsible for your own professional development. But one of your manager’s top responsibilities is to help you identify and advance along the path that is best for you. And performance reviews are a great opportunity to make you think about the future.

Regardless of how your company manages performance, these are the key questions you should think about. Performance feedback is a great opportunity to focus on professional development — your own and that of the people you work with everyday. Make the most of it!

Let me preface this post by saying that this is my personal blog, and that my opinions here are not necessarily those of my employer.

With that out of the way, I love the premise of Hitch.me: a dating site for professionals based on LinkedIn. I won’t confirm or deny the number of my colleagues who have thought about building a dating site based on our data, but it’s great to see someone using our APIs to do so. And the marketing video, while not exactly politically correct, is brilliant.

Yet another reason to work as a data scientist at LinkedIn!

Information Scent

In the early 1990s, PARC researchers Peter Pirolli and Stuart Card developed the theory of information scent (more generally, information foraging) to evaluate user interfaces in terms of how well users can predict which paths will lead them to useful information. Like many HCIR researchers and practitioners, I’ve found this model to be a useful way to think about interactive information seeking systems.

Specifically, faceted search is an exemplary application of the theory of information scent. Faceted search allows users to express an information need as a keyword search, providing them with a series of opportunities to improve the precision of the initial result set by restricting it to results associated with particular facet values.

For example, if I’m looking for folks to hire for my team, I can start my search on LinkedIn with the keywords [information retrieval], restrict my results to Location: San Francisco Bay Area, and then further restrict to School: CMU.

Precision / Recall Asymmetry

Faceted search is a great tool for information seeking systems. But it offers a flow that is asymmetric with respect to precision and recall.

Let’s invert the flow of faceted search. Rather than starting from a large, imprecise result set and progressively narrowing it; let’s start from a small, precise result set and progressively expand it. Since faceted search is often called “guided navigation” (a term Fritz Knabe and I coined at Endeca), let’s call this approach “guided exploration” (which has a nicer ring than “guided expansion”).

Guided exploration exchanges the roles of precision and recall. Faceted search starts with high recall and helps users increase precision while preserving as much recall as possible. In contrast, guided exploration starts with high precision and helps users increase recall while preserving as much precision as possible.

That sounds great in theory, but how can we implement guided exploration in practice?

Let’s remind ourselves why faceted search works so well. Faceted search offers the user information scent: the facet values help the user identify regions of higher precision relative to his or her information need. By selecting a sequence of facet values, the user arrives at a non-empty set that consists entirely or mostly of relevant results.

How to Expand a Result Set

How do we invert this flow? Just as enlarging an image is more complicated than reducing one, increasing recall is more complicated than increasing precision.

If our initial set is the result of selecting multiple facet values, then we may be able to increase recall by de-selecting facet values (e.g., de-selecting San Francisco Bay Area and CMU in my previous example). If we are using hierarchical facets, then rather than de-selecting a facet value, we may be able to replace it with a parent value (e.g., replacing San Francisco Bay Area with California). We can also remove one or more search keywords to broaden the results (e.g., information or retrieval).

Those are straightforward query relaxations. But there are more interesting ways to expand our results:

• We can replace a facet value with the union or that value and similar values (e.g., replacing CMU with CMU OR MIT).
• We can replace the entire query (or any subquery) with a union of that query and the results for selecting a single facet value (e.g., ([information retrieval] AND Location: San Francisco Bay Area AND School: CMU) OR Company: Google)
• We can replace the entire query (or any subquery) with a union of that query and the results for a keyword search a single facet value (e.g., ([information retrieval] AND Location: San Francisco Bay Area AND School: CMU) OR [faceted search]).

As we can see, there are many ways to progressively refine a query in a way that expands the result set. The question is how we provide users with options that  increase recall while preserving as much precision as possible.

Frequency : Recall :: Similarity : Precision

Developers of faceted search systems don’t necessarily invest much thought into deciding which faceted refinement options to present to users. Some systems simply avoid dead ends, offer user all refinement options that least to a non-zero result set. This approach breaks down when there are too many options, in which case most systems offer users the most frequent facet values. A chapter in my faceted search book discusses some other options.

Unfortunately, the number of options for guided exploration – at least if we go beyond the very limited basic options — is too vast to apply such a naive approach. Unions never lead to dead ends, and we don’t have a simple measure like frequency to rank our options.

Or perhaps we do. A good reason to favor frequent values as faceted refinement options is that they tend to preserve recall. What we need is a measure that tends to preserving precision when we expand a result set.

That measure is set similarity. More specifically, it is the asymmetric similarity between a set and a superset containing it, which we can think of as the former’s representativeness of the latter. If we are working with facets, we can measure this similarity in terms of differences between distributions of the facet values. If the current set has high precision, we should favor supersets that are similar to it in order to preserve precision.

I’ll spare readers the math, but I encourage you to read about Kullback-Leibler divergence and Jensen-Shannon divergence if you are not familiar with measures of similarity between probability distributions. I’m also glossing over key implementation details  – such as how to model distributions of facet values as probability distributions, and how to handle  smoothing and normalization for set size. I’ll try to cover these in future posts. But for now, let’s assume that we can measure the similarity between a set and a superset.

Guided Exploration: A General Framework

We now have the elements to put together a general framework for guided exploration:

• Generate a set of candidate expansion options from the current search query using operations such as the following:
• De-select a facet value.
• Replace a facet value with its parent.
• Replace a facet value with the union of it and other values from that facet.
• Remove a search keyword.
• Replace a search keyword with the union of it and related keywords.
• Replace the entire query with the union of it and a related facet value selection.
• Replace the entire query with the union of it and a related keyword search.
• Evaluate each expansion option based on the similarity of the resulting set to the current one.
• Present the most similar sets to the user as expansion options.

Visualizing Drift

It’s one thing to tell a user that two sets are distributionally similar based on an information-theoretic measure, and another to communicate that similarity in a language the user can understand. Here is an example of visualizing the similarity between [information retrieval] AND School: CMU and [information retrieval] AND School: (CMU or MIT):

As we can see from even this basic visualization, replacing CMU with (CMU OR MIT) increases the number of results by 70% while keeping a similar distribution of current companies — the notable exception being people who work for their almae matres.

Conclusion

Faceted search offers some of the most convincing evidence in favor of Gary Marchionini‘s advocacy that we “empower people to explore large-scale information bases but demand that people also take responsibility for this control”. Guided exploration aims to generalize the value proposition of faceted search by inverting the roles of precision and recall. Given the importance of recall, I hope to see progress in this direction. If this is a topic that interests you, give me a shout. Especially if you’re a student looking for an internship this summer!

Every year brings its own adventures, but for me 2011 will be a tough act to follow.

A year ago, I’d just started working at LinkedIn, and my biggest concern was selling our apartment in Brooklyn so that my family could join me in California.

Little did I imagine that my new manager, who had just recruited me from Google to LinkedIn (and persuaded my family to change coasts!), would leave three months later for a startup. Welcome to Silicon Valley! At the time, I felt unready for the abrupt transition into the product executive team. In retrospect, I’m thankful for the kick in the pants that helped me transform my role and brought the best out of a great team.

Summer brought the excitement of LinkedIn’s IPO. The process was exhilarating, especially to someone who had worked for over a decade at a pre-IPO company.

Nonetheless, we didn’t let the IPO distract us from our mission. In March, we celebrated our 100 millionth member; by November, we passed 135 million. And lots more. We released new data products like Skills and Alumni. We won the OSCON Data Innovation Award for contributions to the open source software for big data. We also acquired a few companies, including search engine startup IndexTank. In short, we heeded the two short words on the back of our commemorative IPO t-shirts: “next play”.

Fall was an intense season of conferences. Between CIKM, HCIR, RecSys, Strata, and a talk at CMU, it was a great opportunity to connect and reconnect with researchers and practitioners around the world. I am particularly proud of the success of this year’s HCIR workshop, which showed how much the workshop (now to become a 2-day symposium!) has grown up in five years.

But what capped my year off was seeing Endeca, the company I helped start in 1999, become one of Oracle’s largest acquisitions. Even though it’s been two years since I left, Endeca will always be a core facet of my professional identity. I look forward to great things from all the folks I worked with.

That brings us to 2012, ready to start a new year of adventures. Tough or not, our job is to make every new year more amazing than the previous ones. I’m ready for the challenge, and I hope you are too.

Here’s a teaser of what I have planned:

• My team at LinkedIn is launching into 2012 with a strong focus on derived data quality and relevance. As regular readers know, I see data quality and richer interfaces for information seeking as inseparable concerns. And, speaking of quality, we’re hiring!
• I’ll be speaking at Strata in a couple of months with Claire Hunsaker of Samasource about “Humans, Machines, and the Dimensions of Microwork“. I’m very excited to talk about the intersection of crowdsourcing and data science. And I’ll be joined by three of LinkedIn’s top data scientists: Monica Rogati, Sam Shah, and Pete Skomoroch.
• I’ll be co-chairing the RecSys Industry Track this fall with Yehuda Koren. I’m honored to have the opportunity to work with Yehuda, who was part of the Netflix Grand Prize team and won best paper at RecSys 2011. We’re still putting together the program, but you can look at last year’s program to get an idea of what’s in store.
• I’ll be at the CIKM Industry Event, this time as an invited speaker. CIKM will be take place in Maui this fall and I’m excited about the program that Evgeniy Gabrilovich is putting together for the Industry Event. It will be an all-invited program, just like last year.

I hope you’re also starting 2012 with a fresh sense of purpose. Let’s take a last moment to reflect on a great 2011, and then…NEXT PLAY!

Better yet, you can now, for the first time in the workshop’s history, watch videos of the presentations. Embedded below are videos of Gary Marchionini’s keynote address and of the two paper presentation sessions. Thanks again to Google for being such a gracious host — now online as well as offline!

Keynote

Morning Presentations

Afternoon Presentations

Privacy is the third rail of the cloud. On one hand, the ease of sharing information and the power of analytics have produced extraordinary value for consumers, as well as great business models for companies that serve those consumers. On the other hand, people have good reason to worry about the unintended consequences of over-sharing.

When I attended the O’Reilly Strata New York Conference in September, I had the pleasure to hear and meet Intelius’s Jim Adler talk about being his company’s “accidental chief privacy officer”. Intelius‘s main product is people search — an area that naturally brings up privacy concerns. Especially since Intelius aggregates and publishes information about people from databases of public records, eroding a history of “privacy through difficulty“. Impressed with Jim’s talk at Strata, I persuaded him to deliver a similar talk at LinkedIn, the video of which you can find above. You can also find his slides on SlideShare.

Jim brings nuance to the discussion of privacy — nuance that discussions of online privacy often lack. For example, he responded to the recent controversy about social networks’ “real names” policy with a measured post entitled “Nyms, Pseudonyms, or Anonyms? All of the Above“.

Jim appropriately opened his talk by disclosing a personal example. He shares his name with a more prominent personal injury lawyer who dominates search results for that name, raising the potential of taint by association. Intelius’s core technical problem is to cluster inputs from the sources it aggregates, thus mapping each person to exactly one record in its database.

Jim went on to note that we are at a stage in the privacy debate where we are likely to see more regulation. He makes a few key observations:

• Social norms, which form the basis of our laws and regulations (the notion of a “reasonable expectation of privacy) have changed suddenly, leading to a “privacy vertigo” where suddenly the whole world now feels like a small town.
• Sharing is a gateway from private to public, which often leads to violation of expectations. This problem is not new, but the efficiency of online sharing dramatically amplifies the unintended consequences of sharing. It is crucial that the parties involved in sharing data also have shared expectations around how that data will be used or disclosed.
• We need to distinguish between data use and data access, and not to try to regulate data use with data access regulations. He cites the Fair Credit Reporting Act as one of the most inspired laws of the last 40 years to regulate data use. If you don’t have time to listen to the whole talk, I recommend you jump to 25:12, where he discusses this law in detail.

There’s a lot more in the talk, so I’m not going to try to summarize it all here. I strongly encourage you to check out the video (which includes lengthy Q&A) and the slides. Better yet, let’s use the comments to discuss!

I’ve posted slides and summaries for all ten CIKM 2011 Industry Event presentations:

• Stephen Robertson (Microsoft Research): Why Recall Matters
• John Giannandrea (Google): Freebase – A Rosetta Stone for Entities
• Jeff Hammerbacher (Cloudera): Experiences Evolving a New Analytical Platform: What Works and What’s Missing
• Khalid Al-Kofahi (Thomson Reuters): Combining Advanced Technology and Human Expertise in Legal Research
• Chavdar Botev (LinkedIn): Databus: A System for Timeline-Consistent Low-Latency Change Capture
• Ben Greene (SAP): Large Memory Computers for In-Memory Enterprise Applications
• David Hawking (Funnelback): Search Problems and Solutions in Higher Education
• Ed Chi (Google): Model-Driven Research in Social Computing
• Vanja Josifovski (Yahoo! Research): Toward Deep Understanding of User Behavior on the Web
• Ilya Segalovich (Yandex): Improving Search Quality at Yandex: Current Challenges and Solutions

This post is last in a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

The final talk of the CIKM 2011 Industry Event was a talk from Yandex co-founder and CTO Ilya Segalovich on “Improving Search Quality at Yandex: Current Challenges and Solutions“.

Yandex is the world’s #5 search engine. It dominates the Russian search market, where it has over 64% market share. Ilya focused on three challenges facing Yandex: result diversification, recency-specific ranking, and cross-lingual search.

For result diversification, Ilya focused on queries containing entities without any addition indicators of intent. He asserted that entities offer a strong but incomplete signal of query intent, and in particular that entities often call for suggested query reformulations. The first step in processing such a query is entity categorization. Ilya said that Yandex achieved almost 90% precision using machine learning, and over 95% precision by incorporating manually tuned heuristics. The second step is enumerating possible search intents for the identified category in order to optimize for intent-aware expected reciprocal rank. By diversifying entity queries, Yandex reduced abandonment on popular queries, increased click-through rates, and was able to highlight possible intents in result snippets.

Ilya then talked about the problem of balancing recency and relevance in handling queries about current events. He sees recency ranking as a diversification problem, since a desire for recent content is a kind of query intent. A challenge is managing recency-specific ranking is to predict the recency sensitivity of the user for a given query. Yandex considers factors such as the fraction of results found that are at most 3 days old, the number of news results, spikes in the query stream, lexical cues (e.g., searches for “explosion” or “fire”), and Twitter trending topics. He also referred to a WWW 2006 paper he co-authored on extracting news-related queries from web query logs. The results of these efforts led to measurable improvements in click-based metrics of user happiness.

Ilya talked about a variety of efforts to support cross-lingual search. Russian users enter a significant fraction (about 15%) of non-Russian queries, but many still prefer Russian-language results. For example, a search for a company name return that company’s Russian-language home page if one is available. Yandex implements language personalization by learning a user’s language knowledge and using it as a factor in relevance computation. Yandex also uses machine translation to serve results for Russian-language queries when there are no relevant Russian-language results.

Ilya concluded by pitching the efforts that Yandex is making to participate in and support the broader information retrieval community, including running (and releasing data for) a relevance prediction challenge. It’s great to see a reminder that there is more to web search than Google vs. Bing, and refreshing to see how much Yandex shares its methodology and results with the IR community.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

Those of you who attended the SIGIR 2009 Industry Track had the opportunity to hear Yahoo researcher Vanja Josifovski make an eloquent case for ad retrieval as a new frontier of information retrieval. At the CIKM 2011 Industry Event, Vanja delivered an equally compelling presentation entitled “Toward Deep Understanding of User Behavior: A Biased View of a Practitioner“.

Vanja first offered a vision in which the web of the future will be  your life partner, delivering life-long pervasive personalized experience. Everything will be personalized, and the experience will pervade your entire online experience — from your laptop to your web-enabled toaster.

He then brought us back to the state of personalization today. For search personalization, the low entropy of query intent makes it difficult — or too risky — to significantly outperform the baseline of non-personalized search. In his view, the action today is in content recommendation and ad targeting, where there is high entropy of intent and lots of room for improvement over today’s crude techniques.

How do we achieve these improvements? We need more data, larger scale, and better methods for reasoning about data. In particular, Vanja noted the data we have today — searches, page views, connections, messages, purchases — represents the user’s thin observable state. In contrast, we lack data about the user’s internal state, e.g., is the user jet-lagged or worried about government debt. Vanja said that the only way to get more data is to motivate users by creating value for them with it — i.e., social is give to get.

Of course, we can’t talk about user’s hidden data without thinking about privacy. Vanja asserts that privacy is not dead, but that it’s in hibernation. So far, he argued, we’ve managed with a model of industry self-governance with relatively minor impact from data leaks — specifically as compared to the offline world. But he is apprehensive at the prospect of a major privacy breach inducing legislation that sets back personalization efforts for decades.

Vanja then talked about current personalization methods, including learning relationships among features, dimensionality reduction, and smoothing using external data. He argues that many of the models are mathematically very similar to one another, and it is difficult to analyze the relative merits of the models as opposed to other implementation details of the systems that use them.

Finally, Vanja touched on scale issues. He noted that the MapReduce framework imposes significant restrictions on algorithms used for personalization, and that we need the right abstractions for modeling in parallel environments.

ps. My thanks to Jeff Dalton for live-blogging his notes.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

Given the extraordinary ascent of all things social in today’s online world, we could hardly neglect this theme at the CIKM 2011 Industry Event. We were lucky to have Ed Chi, who recently left the PARC Augmented Social Cognition Group to work on Google+, presenting “Model-Driven Research in Social Computing“.

Ed warned us at the beginning of the talk that his focus would be on work he’d done prior to joining Google. Nonetheless, he offered an interesting collection of public statistics about social activity associated with Google properties: 360M words per day being published on Blogger, 150 years of YouTube video being watched everyday on Facebook, and 40M+ people using Google+. Regardless of how Google has fared in the competition for social networking mindshare, Google is clearly no stranger to online social behavior.

Ed then dove into recent research that he and colleagues have done on Twitter activity. Since all of the papers he discussed are available online, I will only touch on highlights. I encourage you to read the full papers:

• Want to be Retweeted? Large Scale Analytics on Factors Impacting Retweet in Twitter Network
• Tweets from Justin Bieber’s Heart: the Dynamics of the “Location” Field in User Profiles
• Is Twitter a Good Place for Asking Questions? A Characterization Study
• Language Matters in Twitter: A Large Scale Study
• Eddi: Interactive Topic-based Browsing of Social Status Streams
• Short and Tweet: Experiments on Recommending Content from Information Streams
• Speak Little and Well: Recommending Conversations in Online Social Streams

Ed talked at some length about language-dependent behavior on Twitter. For example, tweets in French are more likely to contain URLs than those in English, while tweets in Japanese are less likely (perhaps because the language is more compact relative to Twitter’s 140-character limit?). Tweets in Korean are far more likely to be conversational (i.e., explicitly mentioning or replying to other users) than those in English. These differences remind us to be cautious in generalizing our understanding of online social behavior from the behavior of English-speaking users. Ed also talked about cross-language “brokers” who tweet in multiple languages: he sees these as indicating connection strength between languages, as well as giving us insight to improve cross-­language communication.

Ed then talked about ways to reduce information overload in social streams. These included Eddi, a tool for summarizing social streams, and zerozero88, a closed experiment to produce a personal newspaper from a tweet stream. In analyzing the results of the zerozero88 experiment, Ed and his colleagues found that the most successful recommendation strategy combined users’ self-voting with social voting by their friends of friends. They also found that users wanted both relevance and serendipity — a challenge since the two criteria often compete with one another.

Ed concluded by offering the following design rule: since interaction costs determine number of the people who participate in social activity, get more people into the system by reducing interaction cost. He asserted that this is a key design principle for Google+.

My skepticism about Google’s social efforts is a matter of public record (cf. Social Utility, +/- 25%; Google±?). But hiring Ed Chi was a real coup for Google, and I’m optimistic about what he’ll bring to the Google+ effort.

ps. My thanks to Jeff Dalton for live-blogging his notes.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

One of the recurring themes at the CIKM 2011 Industry Event was that not all search is web search. Stephen Robertson, in advocating why recall matters, noted that web search was exceptional rather than typical as an information retrieval domain. Khalid Al-Kofahi on spoke about the challenges of legal search. Focusing on a different vertical, Funnelback Chief Scientist David Hawking spoke about “Search Problems and Solutions in Higher Education“.

David spent most of the presentation focusing on work that Funnelback did for the Australian National University. Funnelback was originally developed by CSIRO and the ANU under the name Panoptic.

The ANU has a substantial web presence, comprised of hundreds of sites and over a million pages. Like many large sites, it suffers from propagation delay: the most important pages are fresh, but material on the outposts can be stale. Moreover, there is broad diversity of authorship.

The university also has a strong editorial stance for ranking search results: the search engine needs to identify and favor official content. Given the proliferation of unofficial content, it can be a challenge to identify official sites based on signals like incoming link count, click counts, and the use of official style templates.

David described a particular application that Funnelback developed for ANU: a university course finder. The problem is similar to that of ecommerce search and calls for similar solutions, e.g., faceted search, auto-complete, and suggestions of related queries. And, just as in ecommerce, we can evaluate performance in terms of conversion rate.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

Large-scale computation was, not surprisingly, a major theme at the CIKM 2011 Industry Event. Ben Greene, Director of SAP Research Belfast, delivered a presentation on “Large Memory Computers for In-Memory Enterprise Applications“.

Ben started by defining in-memory computing as “technology that allows the processing of massive quantities of real time data in the main memory of the server to provide immediate results from analyses and transactions”. He then asked whether the cloud enables real-time computing, since there is a clear market hunger for cloud computing to solve the problems of our current enterprise systems.

Not surprisingly, he advocated in-memory computing as the solution for those problems. Like John Ousterhout and the RAMCloud team, he sees the need to scale DRAM memory independently from physical boxes. He proposed a model of coherent shared memory, using high-speed low-latency networks and separating the data transport and cache layers into a separate tier below the operating system. The goal: no server-side application caches, DRAM-like latency for physically distributed databases, and in fact no separation between the application server and the database server.

Ben argued that coherent shared memory can dramatically lower the cost of in-memory computing while minimizing the pain for application developers. He also offered some benchmarks for SAP’s BigIron system to demonstrate the performance improvements.

In short, Ben offered a vision of in-memory computing as a reincarnation of the mainframe. It was an interesting and provocative presentation, and my only regret is that we couldn’t stage a debate between him and Jeff Hammerbacher over the future of large-scale enterprise computing.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

I’m of course delighted that one of my colleagues at LinkedIn was able to participate in the CIKM 2011 Industry Event. Principal software engineer Chavdar Botev delivered a presentation on “Databus: A System for Timeline-Consistent Low-Latency Change Capture“.

LinkedIn processes a massive amount of member data and activity. It has over 135M members and is growing faster than two new members per second. Based on recent measurements, those members are on track to perform more than four billion searches on the LinkedIn platform in 2011. All of this activity requires a data change capture mechanism that allows external systems, such as its graph index and real-time full-text search index Zoie, to act as subscribers in user space and stay up to date with constantly changing data in the primary stores.

LinkedIn has built the Databus system to meet these needs. Databus meets four key requirements: timeline consistency, guaranteed delivery, low latency, and user-space visibility. For example, edits to member profile fields, such as companies and job titles, need to be standardized. Also, in order to give recruiters act quickly on feedback to their job postings, we need to be able to propagate the changes to the job description in near-real-time.

Databus propagates data changes throughout LinkedIn’s architecture. When there is a change in a primary store (e.g., member profiles or connections), the changes are buffered in the Databus Relay through a push or pull interface. The relay can also capture the transactional semantics of updates. Clients poll for changes in the relay. If a client falls behind the stream of change events in the relay, it is redirected to a Bootstrap database that delivers a compressed delta of the changes since the last event seen by the client.

In contrast to generic message systems (including the Kafka system that LinkedIn has open-sourced through Apache), Databus has moreinsight in the structure of the messages and can thus do better than just guaranteeing message-level integrity andtransactional semantics for communication sessions.

I tend to live a few levels above core infrastructure, but I’m grateful that Chavdar and his colleagues build the core platform that makes all of our large-scale data collection possible. After all, without data we have no data science.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

The original program for the CIKM 2011 Industry Event featured Peter Jackson, who was chief scientist at Thomson Reuters and author of numerous books and papers on natural language processing. Sadly, Peter died on August 3,2011. Thomson Reuters R&D VP of Research Khalid Al-Kofahi graciously agreed to speak in his place, delivering a presentation on  ”Combining Advanced Search Technology and Human Expertise in Legal Research“.

Khalid began by giving an “83-second” overview of the US legal system, laying out the roles of the law, the courts, and the legislature. He did so to provide the context for the domain that Thomson Reuters serves — namely, legal information. Legal information providers curate legal information, enhance it editorially and algorithmically, and work to make legal information findable and explainable in particular task contexts. He then worked through an example of how a case law document (specifically, Burger King v. Rudzewicz), appears in WestLawNext, with annotations that include headnotes, topic codes, citation data, and historical context.

Channelling William Goffman, Khalid asserted that a document’s content (words, phrases, metadata) are not sufficient to determine its aboutness and importance. Rather, we also have to consider what other people say about the document and how they interact with it. This is especially true in the legal domain because of the precedential nature of law. He then framed legal search in terms of information retrieval metrics, stating the requirements as completeness (recall), accuracy (precision), and authority. Not surprisingly, Khalid agreed with Stephen Robertson’s emphasis on the importance of recall.

Speaking more generally, Khalid noted that vertical search is not just about search. Rather, it’s about findability. which includes navigation, recommendations, clustering, faceted classification, collaboration, etc. Most importantly, it’s about satisfying a set of well-understood tasks. And, particularly in the legal domain, customers demand explainable models. Beyond this demand, explainability serves an additional purpose: it enables the human searcher to add value to the process (cf. human-computer information retrieval).

It is sad to lose a great researcher like Peter Jackson from our ranks, but I am grateful that Khalid was able to honor his memory by presenting their joint work at CIKM. If you’d like to learn more, I encourage you to read the publications on the Thomson Reuters Labs page.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

The third speaker in the program was Cloudera co-founder and Chief Scientist Jeff Hammerbacher. Jeff, recently hailed by Tim O’Reilly as one of the world’s most powerful data scientists, built the Facebook Data Team, which is most known for open-source contributions that include Hive and Cassandra. Jeff’s talk was entitled “Experiences Evolving a New Analytical Platform: What Works and What’s Missing“. I am thankful to Jeff Dalton for live-blogging a summary.

Jeff’s talk was a whirlwind tour through the philosophy and technology for delivering large-scale analytics (aka “big data”) to the world:

1) Philosophy

The true challenges in the task of data mining are creating a data set with the relevant and accurate information and determining the appropriate analysis techniques. While in the past it made sense to plan data storage and structure around the intended use of the data, the economics of storage and the availability of open-source analytics platforms argue for the reverse: data first, ask questions later; store first, establish structure later. The goal is to enable everyone — developers, analysts, business users — to “party on the data”, providing infrastructure that keeps them from clobbering one another or starving each other of resources.

2) Defining the Platform

No one just uses a relational database anymore. For example, consider Microsoft SQL Server. It is actually part of a unified suite that includes SharePoint for collaboration, PowerPivot for OLAP, StreamInsight for complex event processing (CEP), etc. As with the LAMP stack, there is a coherent framework analytical data management which we can call an analytical data platform.

3) Cloudera’s Platform

Cloudera starts with a substrate architecture of Open Compute commodity Linux servers configured using Puppet and Chef and coordinated using ZooKeeper. Naturally this entire stack is open-source. They use HFDS and Ceph to provide distributed, schema-less storage. They offer append-only table storage and metadata using Avro, RCFile, and HCatalog; and mutable table storage and metadata using HBase. For computation, they offer YARN (inter-job scheduling, like Grid Engine, for data intensive computing) and Mesos for cluster resource management; MapReduce, Hamster (MPI), Spark, Dryad / DryadLINQ, Pregel (Giraph), and Dremel as processing frameworks; and Crunch  (like Google’s FlumeJava), PigLatin, HiveQL, and Oozie as high-level interfaces. Finally, Cloudera offers tool access through FUSE, JDBC, and ODBC; and data ingest through Sqoop and Flume.

4) What’s Next?

For the substrate, we can expect support for fat servers with fat pipes, operating system support for isolation, and improved local filesystems (e.g., btrfs). Storage improvements will give us a unified file format, compression, better performance and availability, richer metadata, distributed snapshots, replication across data centers, native client access, and separation of namespace and block management. We will see stabilization of our existing compute tools and better variety, as well as improved fault tolerance, isolation and workload management, low-latency job scheduling, and a unified execution backend for workflow. And we will see better integration through REST API access to all platform components, better document ingest, maintenance of source catalog and provenance information, and an integration beyond ODBC with analytics tools. We will also see tools that facilitate that transition from unstructured to structured data (e.g. RecordBreaker).

Jeff’s talk was as information-dense as this post suggests, and I hope the mostly-academic CIKM audience was not too shell-shocked. It’s fantastic to see practitioners not only building essential tools for research in information and knowledge management, but reaching out to the research community to build bridges. I saw lots of intense conversation after his talk, and I hope the results realize the two-fold mission of the Industry Event, which is to give  researchers an opportunity to learn about the problems most relevant to industry practitioners, and to offer practitioners an opportunity to deepen their understanding of the field in which they are working.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

The second speaker in the program was Metaweb co-founder John Giannadrea. Google acquired Metaweb last year and has kept its promise to to maintain Freebase as a free and open database for the world (including for rival search engine Bing – though I’m not sure if Bing is still using Freebase). John’s talk was entitled “Freebase – A Rosetta Stone for Entities“. I am thankful to Jeff Dalton for live-blogging a summary.

John started by introducing Freebase as a representation of structured objects corresponding to real-world entities and connected by a directed graph of relationships. In other words, a semantic web. While it isn’t quite web-scale, Freebase is a large and growing knowledge base consisting of 25 million entities and 500 million connections — and doubling annually. The core concept in Freebase is a type, and an entity can have many types. For example, Arnold Schwarzenegger is a politician and an actor. John emphasized the messiness of the real world. For example, most actors are people, but what about the dog who played Lassie? It’s important to support exceptions.

The main technical challenge for Freebase is reconciliation — that is, determining how similar a set of data is to existing Freebase topics. John pointed out how critical it is for Freebase to avoid duplication of content, since the utility of Freebase depends on unique nodes in its graph corresponding to unique objects in the world. Freebase obtains many of its entities by reconciling large, open-source knowledge bases — including Wikipedia, WordNet, Library of Congress Authorities,  and metadata from the Stanford Library. Freebase uses a variety of tools to implement reconciliation, including Google Refine (formerly known as Freebase Gridworks) and Matchmaker, a tool for gathering human judgments. While reconciliation is a hard technical problem, it is made possible by making inferences across the web of relationships that link entities to one another.

John then presented Freebase as a Rosetta Stone for entities on the web. Since an entity is simply a collection of keys (one of which is its name), Freebase’s job is to reverse engineer the key-value store that is distributed among the entity’s web references, e.g., the structured databases backing web sites and encoding keys in URL parameters. He noted that Freebase itself is schema-less (it is a graph database), and that even the concept of a type is itself an entity (“Type type is the only type that is an instance of itself”). Google makes Freebase available through an API and the Metaweb Query Language (MQL).

Freebase does have its challenges. The requirement to keep out duplicates is an onerous one, as they discovered when importing a portion of the Open Library catalog. Maintaining quality calls for significant manual curation, and quality varies across the knowledge base. John asserted that Freebase provides 99% accuracy at the 95th percentile, though it’s not clear to me what that means (update: see Bill’s comment below).

While I still have concerns about Freebase’s robustness as a structured knowledge base (see my post on “In Search Of Structure“), I’m excited to see Google investing in structured representations of knowledge. To hear more about Google’s efforts in this space, check out the Strata New York panel I moderated on Entities, Relationships, and Semantics — the panelists included Andrew Hogue, who leads Google’s structured data and information extraction group and managed me during my year at Google New York.

Earlier this year, I had the privilege to moderate a panel at Strata New York 2011 on Entities, Relationships, and Semantics: the State of Structured Search. The four panelists are people I’ve had the pleasure to work with over the years: Andrew Hogue (Google), Breck Baldwin (Alias-i), Evan Sandhaus (New York Times), Wlodek Zadrozny (IBM Research). They work on some of the world’s largest structured search problems — from offering users structured search on Google’s web corpus to building a computing system that defeated Jeopardy! champions in an extreme test of natural language understanding.

O’Reilly has compiled the nearly 50 hours of video from the conference and made the collection available for purchase. I was lucky to attend all of the keynotes and many of the breakout sessions, and I highly recommend them. In the meantime, you can see a recording of the panel I moderated.

Dan Woods has been interviewing a variety of folks to answer the question: “What is a data scientist?“, and I had the honor to participate in his series.

Here is a teaser of my interview:

Above all, a data scientist needs to be able to derive robust conclusions from data. But a data scientist also needs to possess creativity and strong communication skills. Creativity drives the process of hypothesis generation, i.e., picking the right problems to solve that will create value for users and drive business decisions.

Read the rest on Forbes.com. And thanks to Drew Conway for the awesome data science Venn diagram above.

Last week, I had the privilege to present a tutorial at the 5th ACM International Conference on Recommender Systems (RecSys 2011). Given my passion for HCIR and my advocacy for transparency in recommender systems, it shouldn’t surprise regular readers that I focused on both. Unfortunately the tutorial was not recorded, but I hope the slides above prove useful. I also encourage you to take a look at the other tutorials, whose slides are posted on the conference site.

Wednesday evening before the workshop, we were treated to a welcome reception, the first of a few meals provided by Google’s excellent chefs. It was a great opportunity to reconnect with old friends and meet many first-time HCIR attendees.

Thursday started with a scrumptious breakfast that included chilaquiles, coconut fritters, and bacon. Last year’s keynote and this year’s local host Dan Russell pulled all the stops — apparently BigTable is the only Google cafe that serves bacon for breakfast! We then proceeded to a poster boaster session in which each poster presenter had a minute to pitch his or her poster. This session set the tone for the rest of the workshop: concentrated ideas and intense audience engagement.

Then came this year’s keynote, Gary Marchionini. It was a particular treat to have Gary as a keynote, since his lecture on “Toward Human-Computer Information Retrieval” inspired me to conceive the HCIR workshop back in 2007. And Gary delivered the goods. He started with a review of the history of HCIR, including some lesser known figures like Don Hawkins (who was in the audience) , Pauline Cochrane, Richard Marcus, and Charles Meadow.  He brought a few chuckles by citing Nick Belkin (who was present) and Sue Dumais (who was not) as the father and mother of HCIR. Naturally he described some of his own work at the University of North Carolina, including the Open Video, Relation Browser, and ResultsSpace projects.But the highlight of his talk was a graph he presented showing two paths to the same user end-state, one of the paths being a smooth progression and the other being a roller-coaster of ups and down. The question of which one was better drew a wide variety of responses, my favorite being Gene Golovchinsky observing that learning is the friction of the information-seeking process.

We broke for coffee and then came back to the first session of paper presentations. Sofia Athenikos presented a semantic search engine that outperformed IMDB in a user study. Chang Liu explored the effect of task difficulty and domain knowledge on dwell times, finding counterintuitive results (at least for me) regarding the correlation of expertise to dwell time. Jingjing Liu presented research on knowledge examination in multi-session tasks. Then came the lightning talks: Mark Smucker on how users examine and process ranked document lists; Jin Kim on simulating associative browsing; Bill Kules on visualizing the stages of exploratory search; and Michael Cole on user domain knowledge and eye movement patterns during search. Way too much goodness to summarize here — I suggest you read the full papers on the workshop site.

Then came lunch — again in BigTable, but this time with outdoor seating — and the poster session. As always, this it the most interactive part of the day: two hours of non-stop discussion that start over food and end with prying people away from discussions about posters. I was especially proud of LinkedIn’s contributions to the poster session, which covered faceted search log analysis, social navigation, and whether it is time to abandon abandonment.

Then back to the second session of  paper presentations. Luanne Freund talked about document usefulness and genre, finding that genre, besides being hard for users to reliably identify, only matters for tasks that involve doing, deciding, learning; but not for those that involve fact finding or problem solving. Gene Golovchinsky presented work on designing for collaboration in information seeking, previewing the system he used for his challenge entry.  Alyona Medelyan used the Pingar search engine to evaluate how search interface features affect performance on biosciences tasks. Then more lightning talks: Rob Capra analyzing faceted search on mobile devices; Keith Bagley on conceptual mile markers for exploratory search; Xiaojun Yuan on how cognitive styles affect user performance; and Mike Zarro on using social tags and controlled vocabularies as search filters.

Last but not least came the HCIR Challenge:

The HCIR 2011 Challenge focuses on the case where recall is everything – namely, the problem of information availability. The information availability problem arises when the seeker faces uncertainty as to whether the information of interest is available at all. Instances of this problem include some of the highest-value information tasks, such as those facing national security and legal/patent professionals, who might spend hours or days searching to determine whether the desired information exists.

The corpus we will use for the HCIR 2011 Challenge is the CiteSeer digital library of scientific literature. The CiteSeer corpus contains over 750,000 documents and provides rich meta-data about documents, authors, and citations.

There were four entries:

• FreeSearch – Literature Search in a Natural Way
  Claudiu S. Firan, Wolfgang Nejdl, Mihai Georgescu (University of Hanover), and Xinyun Sun (DEKE Lab MOE, Renmin)
  
• Session-based search with Querium
  Gene Golovchinsky (FX Palo Alto Lab) and Abdigani Diriye (University College London)
  
• GisterPro
  David L.Ostby and Edmond Brian (Visual Purple)
  
• Query Analytics Workbench
  Antony Scerri, Matthew Corkum, Keith Gutfreund, Ron Daniel Jr., Michael Taylor (Elsevier Labs)

The competition was fierce. Claudiu showed off the Faceted DBLP interface, which is well suited to the information availability task on CiteSeer data. Ed showed how GisterPro uses visualization to support the information seeking process. But it came down to a close call between the Query Analytics Workbench and Querium. Despite the Elsevier team’s impressive functionality and animated presentation, Gene’s simpler interface and application of ranked fusion won the day. Congratulations to Gene and Abdigani, this year’s HCIR Challenge winners!

We wrapped up the evening at the Tied House, a local microbrewery. And of course the discussion turned to where, when, and how we will hold next year’s workshop. Watch this space. In the meantime, my heartfelt thanks to everyone who made this year’s workshop such a success — and especially to our sponsors. Thank you Endeca, Kent State, Microsoft, and Google!

Today is a wonderful day for Endeca and Oracle! Oracle has announced that it has entered into an agreement to acquire Endeca, bringing together two of the powerhouses of information access. Quoting from the announcement: “The combination of Oracle and Endeca is expected to create a comprehensive technology platform to process, store, manage, search and analyze structured and unstructured information together. ”

As part of Endeca’s founding team, I am very proud to see this day. My ten years at Endeca were a formative experience that established my professional identity and inspired my passion to pursue the vision of human-computer information retrieval (by happy coincidence, the 5th annual HCIR workshop take place on Thursday). Reading Oracle’s presentation about the acquisition, I’m excited to see how Endeca’s technology will play a key role in unifying structured and unstructured data management and analysis for Oracle’s customers.

I take pride in my contributions to Endeca — I still slip sometimes and refer to Endeca as “we”. But the real heroes here are the folks — and especially the leadership —  who have seen this journey through from start to finish. In particular, I am grateful to Steve Papa, Pete Bell, Adam Ferrari, Jack Walter, Keith Johnson, Nik Bates-Haus, and Jason Purcell for everything they have done to bring about this extraordinary outcome.

Finally, excited as I am about this event, it is only the beginning. I am excited to see Endeca’s people and technology powering one of the world’s largest enterprise software companies. Looking forward to the next play!

I had a great audience, including the department head! Of course that meant fielding tough questions, but that’s what makes it fun to present at my alma mater. Now that it’s been over a decade since my defense, I can handle the tough questions.

Unfortunately there is no video, but hopefully the slides are reasonably self-explanatory. If you have questions, please ask them in the comments.

Tonight I’m taking a red-eye to Pittsburgh so that I can spend three days at my (doctoral) alma mater, CMU. In addition to spending time with lots of great students and faculty, my goal is to communicate a taste of the hard computer science problems we are solving (or trying to solve!) at LinkedIn. I’m giving a tech talk Tuesday afternoon, joining my colleagues for an info session Tuesday evening, and participating in the Technical Opportunities Conference (TOC) Wednesday.

Here’s a teaser for my tech talk:

You can find more details about LinkedIn’s visits to CMU and other campuses at http://studentcareers.linkedin.com/.

Hopefully some of you are attending the O’Reilly Strata Conference in New York this Thursday and Friday. If so, I encourage you to attend my panel session on “Entities, Relationships, and Semantics: the State of Structured Search“:

Structured search improves the search experience through the identification of entities and their relationships in documents and queries. This panel will explore the current state of structured and semi-structured search, as well as exploring the open problems in an area that promises to revolutionize information seeking.

The four panelists work on some of the world’s largest structured search problems, from offering users structured search on Google’s web corpus to building a computing system that defeated Jeopardy! champions in an extreme test of natural language understanding. They work on the data, tools, and research that are driving this field. They are all excellent researchers and presenters, promising to offer a informative and engaging panel discussion, for which I will act as moderator.

Panelists:

• Andrew Hogue is a Senior Staff Engineer and Engineering Manager in the Search Quality group at Google New York. He has worked on a wide array of projects including question answering, Google Squared, sentiment analysis, local and product search, and Google Goggles. His is interested in the areas of structured data, information extraction, and machine learning, and their applications to search and search interfaces. Prior to Google, he earned a M.Eng. and B.S. in Computer Science from MIT.

• Breck Baldwin is the President of Alias-i, creators of the popular LingPipe computational linguistics toolkit. He received his Ph.D. in computer science in 1995 from the University of Pennsylvania. In the time between his thesis on coreference resolution and evaluation and founding Alias-i in 1999, Breck worked on DARPA-funded projects through the University of Pennsylvania.

• Evan Sandhaus works as the Semantic Technologist in The New York Times Research and Development Labs. He is spearheading The New York Times Linked Open Data Strategy and overseeing the release of 1.8 million documents to the computer science research community. Previously, Evan helped to put The New York Times on Google Earth, collaborated with New York University to explore new directions in News Search, and worked to bring The New York Times to Facebook.

• Wlodek Zadrozny is an IBM Researcher working on natural language applications. Most recently he worked on text sources for Watson (IBM’s Jeopardy chamption) and applying related DeepQA technology to business problems. His previous work ranged from language processing research to product development and technical planning; in particular, he lead the development of interactions systems that used speech, natural language and focused search. Wlodek Zadrozny received a Ph.D. in Mathematics, from the Polish Academy of Science.

And one more thing. Karaoke at Second on Second in the East Village on Friday night. It’s an unofficial Strata after-party, so come join us Big Data folks for some Big Fun.

I grew up in New York City. On September 11th, 2001, I was in Cambridge, Massachusetts, desperately trying to get through to my parents by all means of communication at my disposal. My dad worked at 40 Worth Street, only a few blocks away from the World Trade Center. Thankfully none of my family or friends were harmed that day, but that fateful event ten years ago left a mark on the world that no one of my generation will ever forget.

Fortunately I have happier associations with this anniversary.

On September 11th, 1999, I boarded an Amtrak from New York to Boston to join Steve Papa, Pete Bell, Dave Gourley, Fritz Knabe, Jack Walter, and Phil Braden to start the company that would eventually be named Endeca. I had no way of knowing whether we would persuade VCs to fund us beyond our six months of seed investment, let alone that we would develop a technology that to revolutionize the search experience of millions of users around the world. Our modest ambition was to build a better way to find stuff on eBay. That goal remains unfulfilled, but 44 of the top 100 online retailers use Endeca, which isn’t too shabby. Especially considering that Endeca has expanded well beyond online retail into domains like manufacturing, business intelligence, and government.

On Seprtember 11th, 2002, I gathered the Endeca founding team for a dinner to celebrate the company’s 3rd birthday. Given my reputation for general irreverence, I feared that my colleagues would think this was a stunt to mock the memory of the more familiar 9/11. But it was quite the opposite. September 11th, 1999 was a turning point in my professional life, and no terrorist was going to take that happiness away from me. To this day I am grateful that my colleagues recognized my sincerity and joined me in this celebration.

The dinner that night was an emotional one: 2002 had been a tough year for the software industry — one in which we saw many of our peer companies fold. Fortunately it was the beginning of much better times for us: from 2003 to 2006, Endeca was the fastest growing private company in Massachusetts. No IPO yet, but the rumors are encouraging.

I left Endeca almost two years ago, going to Google and then LinkedIn. But I will always have fond memories of the decade I spent at Endeca — an experience that established much of the passion that drives me today. I am very proud to have been part of the founding team of such a great company, even if now I can only follow from a distance.

Happy birthday, Endeca, and many more to come!

As many of you know, I’m a proud alumnus of the CMU School of Computer Science (yes, I also attended the CMU of Massachusetts). I’m delighted to have the opportunity to spend a few days on campus this month, and I hope that I’ll have a chance to meet with lots of students and faculty while I’m there.

Specifically, I’ll be giving a talk at Eugene Fink’s Intelligence Seminar on Tuesday, September 20th at 3:30pm in Gates-Hillman 4303:

Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

LinkedIn operates the world’s largest professional network on the Internet with more than 120 million members in over 200 countries. In order to connect its users to the people, opportunities, and content that best advance their careers, LinkedIn has developed a variety of algorithms that surface relevant content, offer personalized recommendations, and establish topic-sensitive reputation — all at a massive scale. In this talk, I will discuss some of the most challenging technical problems we face at LinkedIn, and the approaches we are taking to address them.

I hope to see all of you there! My colleagues and I will also be hosting an information session that same Tuesday at 6pm in Porter Hall, Room 125B, as well as participating in the Technical Opportunities Conference Tuesday and Wednesday. And of course LinkedIn will be conducting on-campus interviews: those will take place all day on Thursday, September 22nd.

If you are a CMU student interested in opportunities at LinkedIn, please apply through TartanTrak (yes, I wish you could just apply with LinkedIn – we’ll get there!). Of course, feel free to reach out to me personally at dtunkelang@linkedin.com. We already have more applicants than slots, but I promise that every application will be considered. I’m very excited to recruit CMU students to strengthen our growing team of software engineers and data scientists.

See you soon, and let’s go Tartans!

A few days ago, our CEO Jeff Weiner led a session at LinkedIn on how to “close” candidates — that is, how to persuade candidates to join your team once you have found and interviewed them. Since not everyone has the opportunity to work at LinkedIn and experience Jeff’s leadership first-hand, I thought I’d share some of his wisdom here.

The key take-away  was that closing a candidate is not about selling the job or company to the candidate, but rather working with the candidate to figure out what the candidate wants and whether the job will help him or her achieve that desire. As an employer, you need to do three things to close a candidate:

1) Figure out what is the candidate’s dream.
2) Determine if job and candidate are the right fit.
3) Communicate your own passion.

Let’s take these one at a time.

Dream.

As I’ve written here in the past, we have to dare to dream. Most of us rely on jobs to sustain us and our loved ones — and for some a job is nothing more than that. There’s no shame in having a dream that is unrelated to a job — Franz Kafka famously worked in a variety of “bread jobs” in order to pay the bills while he wrote novels. Others find their calling as humanitarians, activists, or care givers. It’s easy for many of us to forget that life isn’t always about work.

But the great thing about working in technology is that you can get paid to fulfill your own dream. Look at Larry and Sergey, who set out to organize the world’s information. Or Steve Jobs, whose dream has been to create innovative products. Not everyone is as specific in their dreams or as successful in realizing them, but, as the saying goes, you have to be in it to win it.

Convincing a person to accept a job offer works best when that job brings the person closer to fulfilling his or her dream. My own decisions to go to Google and then LinkedIn are good examples. Working at Endeca drove me to pursue a vision of HCIR — to optimize the way people and machines work together to solve information seeking and exploration tasks. At Google, I hoped to bring exploratory search to the open web. I’ll concede that I did not make much headway, but I’m glad that I tried.

And at LinkedIn, I work on problems that not only stretch the boundaries of information science, but whose solutions help millions of other people achieve their dreams by making them more successful professionally. My dream is to truly reduce HCIR to practice so that people can lead better and more productive lives. Once the folks at LinkedIn understood my dream, closing me was just a matter of offering me the keys to make that dream a reality.

If you want someone to work at your company, get to know that person’s dreams. If the job you are offering can’t help him or her realize those dreams, be honest about it. It’s better for both of you, and for a world that is better off with people devoting their lives’ work to fulfilling their dreams.

Fit.

Fit is a two way street: the candidate should be right for the job, and the job should be right for the candidate. The interviewing process typically focuses on establishing the former, but we often forget that the candidate’s decision focuses on the latter. Just because someone is capable of doing a job doesn’t mean it’s the right job for that person.

For me, fit means many things. A work environment where people work hard and take the company’s success personally. Incentives that allow everyone to win, rather than a zero-sum game where people compete for scarce opportunities. Openness, since I’m someone who lives most of my life in public. I could go on — but I hope you get the general idea. Fit is the set of functional and non-functional requirements that determine whether someone will enjoy a job. And people who enjoy their jobs tend to be productive and stay a while.

If you are trying to persuade someone to accept a job offer, you have to see the decision from that person’s point of view. In other words, ask yourself — and convincingly answer — why the job is the right fit for the candidate. That means accepting the possibility that is isn’t the right fit, and doing right by the candidate even if that means backing off.

Passion.

Choosing a job is one of the most important life decisions that people make. It’s not quite up there with getting married or having a child, but it’s a a decision that most people take (and should take) very seriously. Some people create spreadsheets of the pros and cons to compare opportunities and try to frame their decision as an optimization problem. Others go with their gut.

Those who know me personally — whether from face-to-face or online interaction — know that I wear my passion on my sleeve. I can’t understand how someone could get up in the morning and go to work without being passionate about his or her job. I know that many people don’t have a choice in the matter, and I pity them. In a country where most people take subsistence for granted, having a job you love strikes me as a necessity, rather than a luxury.

But what is clear is that if you, as an employer, are not passionate about what you do, you have no business expecting a candidate to take such a big leap of faith with you. Moreover, passion is hard to fake. As it should be — I’m not suggesting that employers should pretend to be excited about their jobs. Rather, your own sincere excitement is a baseline for those you hope to attract to your team. Passion is contagious, and passion is the raw material for making dreams come true.

Dream. Fit. Passion.

There you have it: dream, fit, passion. And remember, closing isn’t selling. Do right by the people you try to hire. After all, jobs are short, but careers are long. Celebrate everyone’s professional success, and take your losses in stride. I can tell you from experience that it all works out for the best.

Interviewing software engineers is hard. Jeff Atwood bemoans how difficult it is to find candidates who can write code. The tech press sporadically publishes “best” interview questions that make me cringe — though I love the IKEA question. Startups like Codility and Interview Street see this challenge as an opportunity, offering hiring managers the prospect of outsourcing their coding interviews. Meanwhile, Diego Basch and others are urging us to stop subjecting candidates to whiteboard coding exercises.

I don’t have a silver bullet to offer. I agree that IQ tests and gotcha questions are a terrible way to assess software engineering candidates. At best, they test only one desirable attribute; at worst, they are a crapshoot as to whether a candidate has seen a similar problem or stumbles into the key insight. Coding questions are a much better tool for assessing people whose day job will be coding, but conventional interviews — whether by phone or in person — are a suboptimal way to test coding strength. Also, it’s not clear whether a coding question should assess problem-solving, pure translation of a solution into working code, or both.

In the face of all of these challenges, I came up with an interview problem that has served me and others well for a few years at Endeca, Google, and LinkedIn. It is with a heavy heart that I retire it, for reasons I’ll discuss at the end of the post. But first let me describe the problem and explain why it has been so effective.

The Problem

I call it the “word break” problem and describe it as follows:

Given an input string and a dictionary of words,
segment the input string into a space-separated
sequence of dictionary words if possible. For
example, if the input string is "applepie" and
dictionary contains a standard set of English words,
then we would return the string "apple pie" as output.

Note that I’ve deliberately left some aspects of this problem vague or underspecified, giving the candidate an opportunity to flesh them out. Here are examples of questions a candidate might ask, and how I would answer them:

Q: What if the input string is already a word in the
   dictionary?
A: A single word is a special case of a space-separated
   sequence of words.

Q: Should I only consider segmentations into two words?
A: No, but start with that case if it's easier.

Q: What if the input string cannot be segmented into a
   sequence of words in the dictionary?
A: Then return null or something equivalent.

Q: What about stemming, spelling correction, etc.?
A: Just segment the exact input string into a sequence
   of exact words in the dictionary.

Q: What if there are multiple valid segmentations?
A: Just return any valid segmentation if there is one.

Q: I'm thinking of implementing the dictionary as a
   trie, suffix tree, Fibonacci heap, ...
A: You don't need to implement the dictionary. Just
   assume access to a reasonable implementation.

Q: What operations does the dictionary support?
A: Exact string lookup. That's all you need.

Q: How big is the dictionary?
A: Assume it's much bigger than the input string,
   but that it fits in memory.

Seeing how a candidate negotiates these details is instructive: it offers you a sense of the candidate’s communication skills and attention to detail, not to mention the candidate’s basic understanding of data structures and algorithms.

A FizzBuzz Solution

Enough with the problem specification and on to the solution. Some candidates start with the simplified version of the problem that only considers segmentations into two words. I consider this a FizzBuzz problem, and I expect any competent software engineer to produce the equivalent of the following in their programming language of choice. I’ll use Java in my example solutions.

String SegmentString(String input, Set<String> dict) {
  int len = input.length();
  for (int i = 1; i < len; i++) {
    String prefix = input.substring(0, i);
    if (dict.contains(prefix)) {
      String suffix = input.substring(i, len);
      if (dict.contains(suffix)) {
        return prefix + " " + suffix;
      }
    }
  }
  return null;
}

I have interviewed candidates who could not produce the above — including candidates who had passed a technical phone screen at Google. As Jeff Atwood says, FizzBuzz problems are a great way to keep interviewers from wasting their time interviewing programmers who can’t program.

A General Solution

Of course, the more interesting problem is the general case, where the input string may be segmented into any number of dictionary words. There are a number of ways to approach this problem, but the most straightforward is recursive backtracking. Here is a typical solution that builds on the previous one:

String SegmentString(String input, Set<String> dict) {
  if (dict.contains(input)) return input;
  int len = input.length();
  for (int i = 1; i < len; i++) {
    String prefix = input.substring(0, i);
    if (dict.contains(prefix)) {
      String suffix = input.substring(i, len);
      String segSuffix = SegmentString(suffix, dict);
      if (segSuffix != null) {
        return prefix + " " + segSuffix;
      }
    }
  }
  return null;
}

Many candidates for software engineering positions cannot come up with the above or an equivalent (e.g., a solution that uses an explicit stack) in half an hour. I’m sure that many of them are competent and productive. But I would not hire them to work on information retrieval or machine learning problems, especially at a company that delivers search functionality on a massive scale.

Analyzing the Running Time

But wait, there’s more! When a candidate does arrive at a solution like the above, I ask for an big O analysis of its worst-case running time as a function of n, the length of the input string. I’ve heard candidates respond with everything from O(n) to O(n!).

I typically offer the following hint:

Consider a pathological dictionary containing the words
"a", "aa", "aaa", ..., i.e., words composed solely of
the letter 'a'. What happens when the input string is a
sequence of n-1 'a's followed by a 'b'?

Hopefully the candidate can figure out that the recursive backtracking solution will explore every possible segmentation of this input string, which reduces the analysis to determine the number of possible segmentations. I leave it as an exercise to the reader (with this hint) to determine that this number is O(2ⁿ).

An Efficient Solution

If a candidate gets this far, I ask if it is possible to do better than O(2ⁿ). Most candidates realize this is a loaded question, and strong ones recognize the opportunity to apply dynamic programming or memoization. Here is a solution using memoization:

Map<String, String> memoized;

String SegmentString(String input, Set<String> dict) {
  if (dict.contains(input)) return input;
  if (memoized.containsKey(input) {
    return memoized.get(input);
  }
  int len = input.length();
  for (int i = 1; i < len; i++) {
    String prefix = input.substring(0, i);
    if (dict.contains(prefix)) {
      String suffix = input.substring(i, len);
      String segSuffix = SegmentString(suffix, dict);
      if (segSuffix != null) {
        memoized.put(input, prefix + " " + segSuffix);
        return prefix + " " + segSuffix;

}
}
memoized.put(input, null);
return null;
}
Again the candidate should be able to perform the worst-case analysis. The key insight is that SegmentString is only called on suffixes of the original input string, and that there are only O(n) suffixes. I leave as an exercise to the reader to determine that the worst-case running time of the memoized solution above is O(n²), assuming that the substring operation only requires constant time (a discussion which itself makes for an interesting tangent).

Why I Love This Problem

There are lots of reasons I love this problem. I'll enumerate a few:

• It is a real problem that came up in the couse of developing production software. I developed Endeca's original implementation for rewriting search queries, and this problem came up in the context of spelling correction and thesaurus expansion.
• It does not require any specialized knowledge -- just strings, sets, maps, recursion, and a simple application of dynamic programming / memoization. Basics that are covered in a first- or second-year undergraduate course in computer science.
• The code is non-trivial but compact enough to use under the tight conditions of a 45-minute interview, whether in person or over the phone using a tool like Collabedit.
• The problem is challenging, but it isn't a gotcha problem. Rather, it requires a methodical analysis of the problem and the application of basic computer science tools.
• The candidate's performance on the problem isn't binary. The worst candidates don't even manage to implement the fizzbuzz solution in 45 minutes. The best implement a memoized solution in 10 minutes, allowing you to make the problem even more interesting, e.g., asking how they would handle a dictionary too large to fit in main memory. Most candidates perform somewhere in the middle.

Happy Retirement

Unfortunately, all good things come to an end. I recently discovered that a candidate posted this problem on Glassdoor. The solution posted there hardly goes into the level of detail I've provided in this post, but I decided that a problem this good deserved to retire in style.

It's hard to come up with good interview problems, and it's also hard to keep secrets. The secret may be to keep fewer secrets. An ideal interview question is one for which advance knowledge has limited value. I'm working with my colleagues on such an approach. Naturally, I'll share more if and when we deploy it.

In the mean time, I hope that everyone who experienced the word break problem appreciated it as a worthy test of their skills. No problem is perfect, nor can performance on a single interview question ever be a perfect predictor of how well a candidate will perform as an engineer. Still, this one was pretty good, and I know that a bunch of us will miss it.

There are four outstanding information retrieval conferences coming up, and I will have the pleasure of participating in three of them. I’d like to make sure readers here are aware of all of them.

The first is KDD 2011, which will take place August 21-24, 2011 in San Diego, CA. The annual ACM SIGKDD conference is the premier international forum for data mining researchers and practitioners from academia, industry, and government to share their ideas, research results and experiences. KDD-2011 will feature keynote presentations, oral paper presentations, poster sessions, workshops, tutorials, panels, exhibits, demonstrations, and the KDD Cup competition. KDD-2011 will run from August 21-24 in San Diego, CA and will feature hundreds of practitioners and academic data miners converging on the one location.

I will not be attending KDD myself, but several of my colleagues will be there. In particular, Ron Bekkerman will be presenting a paper on “High-Precision Phrase-Based Document Classification on a Modern Scale”, as well as offering a tutorial on “Scaling Up Machine Learning: Parallel and Distributed Approaches”.

Workshop on Human-Computer Interaction and Information Retrieval (HCIR 2011) - Mountain View, CA – October 20, 2011

The second is HCIR 2011, the fifth annual HCIR workshop, which I am co-organizing. It will be held all day on Thursday, October 20th, 2011 at Google’s main campus in Mountain View, California. There will be a reception on Wednesday evening before the workshop. Our keynote speaker this year will be Gary Marchionini, Dean of the School of Information and Library Science, University of North Carolina at Chapel Hill. We are also excited to continue the HCIR Challenge, this year focusing on the problem of information availability, where the seeker faces uncertainty as to whether the information of interest is available at all. The corpus will be the CiteSeer digital library of scientific literature, which contains over 750,000 documents and provides rich meta-data about documents, authors, and citations.

Thanks to generous contributions made by Google, Microsoft Research, and Endeca, there will be no registration fee for HCIR this year. Information about how to register will be sent to authors of accepted position papers, research papers, and challenge reports. Note that the submission deadline has been extended by two weeks to Sunday, August 14th. I strongly encourage you to submit in one of these categories in you are working in this field.

The third is RecSys 2011, the 5th ACM International Conference on Recommender Systems. RecSys 2011 builds on the success of the Recommenders 06 Summer School in Bilbao, Spain and the series of four successful conference events from 2007 to 2010 in Minneapolis (2007), Lausanne (2008), New York (2009) and Barcelona (2010). In these events many members of the practitioner and research communities valued the rich exchange of ideas made possible by the shared plenary sessions. The 5th International conference will promote the same close interaction among practitioners and researchers.

I will be giving a tutorial at RecSys 2011 on “Recommendations as a Conversation with the User”.

The fourth is CIKM 2011, the 20th ACM Conference on Information and Knowledge Management. It will take place in Glasgow, Scotland, UK, 24th-28th October 2011. Since 1992, the CIKM has successfully brought together leading researchers and developers from the database, information retrieval, and knowledge management communities. The purpose of the conference is to identify challenging problems facing the development of future knowledge and information systems, and to shape future research directions through the publication of high quality, applied and theoretical research findings. CIKM 2011 will continue the tradition of promoting collaboration among multiple areas in the general areas of databases, information retrieval, and knowledge management.

I am proud to be organizing the CIKM 2011 Industry Event, which will feature such industry heavyweights as Stephen Robertson (Microsoft Research), John Giannandrea (Google), Vanja Josifovski (Yahoo! Research), Ilya Segalovich (Yandex), Jeff Hammerbacher (Cloudera), and Chavdar Botev (LinkedIn).

I’m very excited about all four of these opportunities to exchange ideas about information retrieval and related areas, and I am grateful to LinkedIn for supporting my participation, as well as that of my colleagues. I hope to see some of you at these events!

A major feature of the recently released Google+ is Circles, which allows you to “share relevant content with the right people, and follow content posted by people you find interesting.”

Most people seem to look at Circles as a privacy feature — and indeed Google’s official description gives the impression that Circles exist to manage privacy based on real-life social contexts. Of course, re-sharing can result in unintended consequences, and Google even offers a warning that:

Unless you disable reshares, anything you share (either publicly or with your circles) can be reshared beyond the original people you shared the content with. This could happen either through reshares or through mentions in comments.

Privacy is a big deal, especially for Google — and particularly in the context of rolling out a new social network. Still, I’m not persuaded that privacy is the only or even the primary concern motivating the concept of social circles.

Sharing content with someone is not just about giving that person permission to see it. Sharing content with someone asserts a claim on that person’s attention. While it may be a privilege for me to have access to your content, it may be even more of a privilege for you that I allocate my scarce attention to consume it.

What if we focus on routing content to the people who would find it most interesting? Such an approach works best if all of the shared content is public with respect to permissions — that is, people post it without any expectation of privacy. Twitter demonstrates that many people are comfortable with such a sharing model. Imagine if they could learn to trust a system that optimizes (or at least attempts to optimize) the allocation of everyone’s attention. This is not an easy problem by any means, nor is it one that is likely to be solved by algorithms alone. It will take a strong dose of HCIR to get it right. But, at least in my view, optimizing the allocation of human attention is the grand challenge that everyone working with information retrieval or social networks should be striving to address.

Privacy is important, and social networks should offer simple, robust privacy controls that users understand. We all have experienced the problem of filter failure. But sharing isn’t just about privacy. Our attention is our most precious cognitive asset, both as individuals and as a society, Moreover, our attention faces ever-increasing demands as our social lives evolve in an online world relatively free of physical constraints. Social network developers would do well to pay attention…to attention.

Diego Basch is the CEO and founder of IndexTank, a hosted search service that powers major web sites such as Reddit, Twitvid, blip.tv, as well as providing a WordPress plug-in for blogs (like this one). Diego gained his search experience working with Inktomi, where he wrote some of the world’s first web-scale link analysis algorithms. He is on a mission to make every search box blazing fast and useful.

So much brainpower is spent solving the wrong problems. The world is filled with solutions looking for problems that nobody has — as illustrated by a Google query for [stupidest inventions ever]. More often, people focus narrowly on a particular approach when they should focus on the problem the approach is intended to solve. Or they take a solution for one problem and assume it will apply to another.

Consider the emphasis that search engine developers place on relevance ranking. It is not hard to understand why web-scale search engines emphasize relevance. For example, a search on Google for [emergency locksmith] returns tens of billions of web pages, among which there are only a handful results that you want. Google must filter out the growing number of lead generation companies that spend a ton of money trying to game its results.

Most web and application developers are familiar with the concept of relevance, so they naturally assume that it should be the primary concern when they add search to their own sites or apps. When I talk to people who want full-text search for their 40,000 book titles or 100k classified ads, they ask me about all the ways they can tune relevance. But often they are focusing on a solution, rather than their fundamental problem.

Developers are (or should be!) trying to improve the user experience of their application search. Too often they wrongly assume that relevance is the single most important factor for optimizing this user experience. Let’s surface this confusion in a concrete example.

As a rock climber, once in a while I feel the aches and pains caused by the sport. As the years go by it’s very important to keep your tendons healthy if you do not want to take forced breaks (or type with one hand!). Rockclimbing.com is one of the most popular climbing sites, and I know some medical professionals who occasionally answer health-related questions there. Let’s search there for [tendon injury prevention].

In the above example, part of the problem is that the search results do not have contextual snippets. Maybe there is relevant information hiding behind a click, but the user has no way of knowing. More generally, there’s no hint as to what results could be better. Information such as score of the answer (which is available), the author’s bio (e.g. “climber, physical therapist”) would make the decision easier. If you need to click and scroll, search within the page, go back and try something else, then the search engine is wasting your time. 

Which brings us the broader point: when users search, they want to spend the least amount of time possible getting to the information they want. Relevance is a means to this end. In particular, clicks and typing costs users time. That time can come from page load, rendering, repeated use of the back button, and of course typing (and re-typing) search queries.

Some application search engines really nail the user experience. Let’s say we’re looking for the movie Koyaanits-however-you-spell-it. Go to the Internet Movie Database (IMDB) and start typing k-o-y-e — and there it is, as the second result. Notice that there is a ton of irrelevant stuff around it but it doesn’t matter. I see what I want very quickly.

Hopefully these two examples serve to illustrate the broader point: search engines should not focus on relevance as an end in itself, but rather on whatever helps users find the information they want as quickly as possible. That means offering contextual snippets, instant feedback, and of course snappy response times. Give users speed, and you will make them happy.

When I left Google last December, it was an open secret that Google was developing a social networking product. Now that Google has released Google+, I am at liberty to share my personal impressions.

Let’s start with the clear wins.

• Impressive launch. Google has certainly learned its lesson from the past launches of Wave and Buzz. Google+ is unambiguously opt-in — no one is going to complain about being ambushed. People have been begging for invites. But Google is wisely releasing invites quickly enough to build critical mass. I’d say that Google has at least picked up the Quora crowd of early adopters in Silicon Valley.

• Clean design. Design lead Andy Hertzfeld (of Macintosh fame) has nailed it, leading bloggers to comment that this looks too well designed to be a Google product. Comparing Google+ to Facebook now, I’m reminded at least a little of comparisons between Facebook and Myspace. Great move for Google here.

Now let’s talk about Google’s three big features here: Circles, Sparks, and Hangouts.

• Circles. Straight out of Paul Adams’s presentation of social networking (which he created before he left Google for Facebook), the idea is simple: a person doesn’t have a single group of friends, but rather several groups that tend are mostly disjoint. Through Circles, Google+ makes this soft partitioning of the social space a core design principle. You add people to one or more circles, follow the stream of activity from a circle, and share with circles. It’s great in theory. But in practice it creates friction, especially for people trained on Facebook. There’s a trade-off between simplicity and expressive power, and Google is placing a strong bet on how users will make this trade-off.  I’m inclined to agree with Yishan Wong that “the sorting of friends into buckets (friend lists) is something that only nerds do”. Given Google’s deep expertise in machine learning, I’m expecting Google to reduce this friction by give users intelligent suggestions. Full disclosure: my colleagues at LinkedIn built InMaps, which infers communities from your social network.

• Sparks. The tagline for Sparks is “For nerding out. Together.” It feels like a positioning designed by Googlers for Googlers– you can see promotional videos here and here. I haven’t seen much talk about Sparks, and what little commentary I’ve seen is less than gushing. I’ve experimented with it a bit from a consumption side, and I confess I’m underwhelmed. Perhaps it’s a chicken-and-egg problem — Sparks will only be useful if users populate their profiles with interests, but right now users have no incentive to do so. If Sparks is Google’s attempt to make Reader more social, there’s still a ways to go. Full disclosure: LinkedIn has its own approach to social news, LinkedIn Today, which seems to be doing something right.  

• Hangouts. In plain English, Hangouts are group video chat embedded in a social network. Which sounds a lot like what Facebook is rumored to be releasing this week through a partnership with Skype. Which in turn was just acquired by Microsoft. Will Apple join the party too by implementing group chat in FaceTime? Competitive dynamics aside, this is a very cool feature that hopefully won’t devolve into Chatroulette. Nothing to, um, disclose here.

But the $64B question is whether all this will matter. Can Google+ sustainably co-exist with Facebook? Will people use both services — and, if so, how will they allocate their attention between them? Or is the success of Google+ predicated on displacing Facebook? Or Twitter? Either of those would certainly qualify as a Big Hairy Audacious Goal.

Like Fred Wilson, I’m rooting for Google+ to succeed — but even Fred notes that he would not be able to get his family on Google+, as they are already happy with Facebook. It’s not clear to me what I can get *today* from Google+ that I can’t get from Facebook.

Granted, I’m not a heavy Facebook user, so I’m not the best person to ask this question. So readers, I ask you: why will or won’t you use Google+?

Don’t worry, it’s safe for work. Well, unless you work for Linden Lab.

Given that I work for the world’s largest professional network, I take work very personally. I’m also deeply involved in LinkedIn’s hiring process, which gives me opportunities to see how people make career decisions. I thought I’d share my own perspective here.

For me there are three things that matter to me about my work:

1. Do I love the work I do? Does work feel like play, stimulating me intellectually and emotionally? Am I excited about the people I work with? Is work a grind, or is it something I do for fun?
2. Is the work I do of value to my employer? Am I justifying my employer’s investment in me, or am I a freeloader lost in the inefficiency of corporate bureaucracy?
3. Is my work making the world a better place? Specifically, is the work I do making the world by more like the world I want to live in?

Not everyone may share my above values, and in any case not every job can address all of these values. But I am fortunate to have found one that does, and I’m loving it. To borrow a phrase, it just works.

If you haven’t seen this video by Dan Pink on what motivates people, I urge you to watch it. It’s a great reminder that there is more to motivation than economic incentives.

Finally, I hope that you are doing work that fulfills you. As I work to grow my great team at LinkedIn, my mission is not only to to bring great people to LinkedIn, but bring great work and fulfillment to great people. Whatever you do, be amazing.

Last weekend I had the extraordinary privilege to attend Foo Camp, an annual gathering of about 250 Friends Of O’Reilly (aka Foo). Tim O’Reilly, Sara Winge, and their colleagues have amazing friends, as you can see if you scan this unofficial list of attendees working on big data, open government, computer security, and more generally on the cutting edge of technology and culture (especially where the two overlap).

Foo Camp is an unconference, which merits some elaboration. No fees, no conference hotel (many attendees literally set up camp in the space O’Reilly provided), and no advance program aside from some preselected 5-minute Ignite presentations. Attendees proposed and organized sessions, merging and re-arranging them to optimize for participation. It was a bit chaotic (especially the mad rush after dinner to secure session slots), but very effective.

The minimalist format brought out the best in participants.

For example, I am passionate about (i.e., against) software patents, so I organized a session about them. I did a double-take when I realized that one of the participants was Pamela Samuelson, perhaps the world’s top expers on intellectual property law. I braced myself to be schooled — as I was. But she did it gently and constructively. Specifically, she pointed me to work that her colleagues Jason Schultz and Jennifer Urban were doing on a defensive patent strategy for open-source software (including a proposed license), as well as reminding me of the Berkeley Patent Survey supporting the argument that software entrepreneurs only file for patents because of real or perceived pressure from their investors. I also heard war stories from lawyers who have done pro bono work against patent trolls, reinforcing my own resolve and also reassuring me that the examples I’ve seen at close range are not isolated.

Another session asked whether we are too data driven in our work. What was notable is that this session included participants from some of the largest internet companies debating some of the must fundamental ways in which we work, e.g., do we actually learn from data or do we engage in assault by data to defend preconceived positions (cf. argumentative theory). Like all of the conference, the discussion was under “frieNDA”. so I’m being intentionally vague on the specifics. But it was refreshing to see candid admission that all of us know and have experienced the dangers of manipulating an audience with data, and that there are no algorithms to enforce common sense and good faith.

I won’t even try to enumerate the sessions and side conversations that excited me — topics included privacy, the future of publishing, a critical analysis of geek culture, and irrational user behavior. I missed the session on data-driven parenting, though others have pointed out to me that you can only learn so much if you don’t have twins and perform A/B tests. The best summary is intellectual diversity and overstimulation. If you’d like to get a general sense of the discussion, check out the #foocamp tweet stream. I also recommend Scott Berkun’s post on “What I learned at FOO Camp“.

As someone who organizes the occasional event, I’m intrigued by the unconference approach — especially now that I’ve experienced it first-hand. Moreover, I feel strongly that the academic conference model needs an upgrade. But I also know that open-ended, free-form discussion sessions are not a viable alternative — indeed, a big part of Foo Camp’s success was how it inspired participants to organize sessions — and to vote with their feet to attend the worthwhile ones. And of course part of that success came from inviting active, engaged participants rather than passive spectators.

Many of you also organize events, and I’m sure that all of you attend them. I’m curious to hear your thoughts about how to make them better, and happy to share more of what I learned at Foo Camp. After all, Foo is for (inspiring) thought.

As promised, here is a video of CMU professor Christos Faloutsos‘s recent tech talk at LinkedIn on “Mining Billion-Node Graphs“. Enjoy!

And check out our next week’s open tech talk by Sreenivas Gollapudi of Microsoft Research on “A Framework for Result Diversification in Search“.

ps. If you like these topics, then please talk to me about opportunities at LinkedIn! My group is hiring, as are many others.

The war for talent. It’s the latest metaphor for the challenge that tech companies face as excitement is building in Silicon Valley again. Well, not really — McKinsey coined the phrase in 1997 and used it as the title of a book published four years later.

But anyone who has been trying to hire great software engineers in recent months knows how hard it is to do so. Particularly for folks like me who are trying to hire data scientists — apparently there’s a national shortage. This is nothing new — as Joel Spolsky noted in a 2006 post, “the great software developers, indeed, the best people in every field, are quite simply never on the market.”

I’m not an expert (or ninja) on the subject of recruiting or employer branding in general, but I’ve seen enough of how companies go about hiring software engineers to know that we can do better. I’d like to share some of my thoughts and experiences, and I hope that you will reciprocate and share your thoughts in the comments. I’m especially interesting in hearing from folks who are at universities (aka hunting grounds) or who are involved in organizing academic conferences.

First, let’s talk about how we measure success. As Lord Kelvin famously said, “If you can’t measure it, you can’t improve it.” I’m not going to talk about how to handle active candidates — that’s a filtering problem which, in my opinion, is much more tractable. For example, see what Joel has to say about interviewing developers. Rather, I’m concerned with the challenge of discovering qualified passive candidates and converting them into active ones. Hence, I propose we make our metric the number of qualified applicants.

The baseline strategy is sourcing, i.e. have sourcers or hiring managers scour the world for qualified candidates (there’s an app for that), entice them with your best recruiting pitch, and then go hog wild on the folks who respond. The success of this strategy depends mainly on the rate at which you, your sourcers, or your hiring managers find qualified candidates — which in turn may split into the two subtasks of finding candidates and filtering them — and the conversion rate for the qualified candidates you find. Since the best candidates are often happy in their current positions, sourcing passive candidates requires a lot of work and a thick skin for rejection.

What are other ways to attract qualified passive candidates? Here are a few, with examples from my experience at LinkedIn:

• Hosting events. Last week at LinkedIn, we hosted CMU professor Christos Faloutsos, who delivered a fantastic talk on “Mining Billion Node Graphs” — a topic we thought interesting enough to justify opening up the talk to the general public. We had a few hundred guests, many of whom are precisely the kinds of folks we are trying to hire. Even more people watched the live stream online or will watch the video when we post it to YouTube (coming soon — stay tuned!). While this was not a recruiting event (we did not even announce that we are hiring), it was a great opportunity to associate LinkedIn with the hard computer science problems we solve on a daily basis.
• Sponsoring events. Sponsorship is tricky — if you’re not careful, you spend a lot of money for a glorified display ad. Sometimes sponsorship offers speaking slots as part of the package, but audiences are rightfully skeptical of speakers who have paid for their slots — especially at conferences that charge hefty fees for attendance. But sometimes sponsorship works. For example, LinkedIn’s was a sponsor of the O’Reilly Strata Conference, and the perks of sponsorship complemented our earned speaker slots, helping us bring enormous visibility to our data scientist team and its recent innovations like InMaps (we has a booth there to print attendees’ InMaps) and Skills (which launched during the conference). While Strata generated few direct leads, it left a lasting impression in the big data community, and I regularly hear candidates refer to it.
• Participating in events. As the Beatles tell us, money can’t buy you love. If you want to make an (positive) impression at a conference, you have to contribute people and ideas. This is especially true at academic conferences, where attendees quickly throw out the the extra weight in their tote bags and focus on the conference’s content and professional networking opportunities. It’s great if you are Microsoft with a team of close to a thousand researchers and can dominate a conference like SIGIR. But smaller companies can still make a strong impression on researchers — and especially on students who may be looking for internships or full-time positions — by taking an active role at conferences. The traditional approach is to submit papers to the main conference track — but other avenues include tutorials, workshops, and industry events. Such participation is often invited, but such invitations are in turn earned by cultivating relationships with researchers — especially the ones who find themselves on organizing committees.
• Contribute to open source projects. The Search, Network, and Analytics (SNA) team at LinkedIn contributes frequently to open-source projects and publicizes some of its work at http://sna-projects.com/. Open source projects are a great way to earn the respect of engineers who value source over PowerPoint. Especially when your employees include committers to key technologies like Hadoop. Moreover, open-source projects are social communities, so contributing to them offers opportunities for employees to interact with potential hires.
• Social media. By now, I’d like to think that marketers understand social media to simply be another set of marketing channels. But I think the territory is still pretty new for employers. Here is a simple suggestion: encourage (but do not try to force) employees to express themselves professionally online. Enforce the standard non-disclosure rules, of course, but don’t try to manage their voices. Authenticity speaks for itself — for example, look at what Adam Nash says about LinkedIn on his personal blog. Or my own posts here. Engineers don’t read press releases or  corporate blogs, but they do pay attention to their peers. And there’s nothing unique about blogs — the same principle applies to platforms like Twitter, Facebook, Quora, and of course LinkedIn. Not all employees enjoy being online extroverts, but those that do not only act as brand ambassadors, but also are likely to eventually strike up conversations with passive candidates about employment opportunities.

Finally, don’t forget measure the results of these efforts! Some activities generate leads directly, in which case you can make an apples-to-apples comparison of their results and costs with the baseline strategy of sourcing. It’s harder to measure the longer-term effect of efforts to raise visibility, but you can at least ask candidates if they are aware of those efforts — after all, efforts to raise visibility should be visible to candidates! You can also ask candidates if those efforts were a factor in their decision to apply. These measures aren’t perfect, but they are a lot better than nothing, especially when you’re trying to decide how best to invest limited resources.

Of course, even an optimal strategy can’t substitute for offering a combination of interesting work, competitive compensation, and a work hard / play hard culture. As with all marketing efforts, you need to start with a great product. But great products don’t sell themselves: you need to invest in a combination of outbound and inbound marketing to have a fighting chance in the war for talent. Good luck! And, in case you didn’t notice, we’re hiring!

If you Google [relevance theory], you’ll discover this Wikipedia entry about a theory proposed by Dan Sperber and Deirdre Wilson arguing that, in any given communication situation, the listener will stop processing as soon as he or she has found meaning that fits his or her expectation of relevance. The Wikipedia entry offers the following example of this principle:

Mary: Would you like to come for a run?

Bill: I’m resting today.

We understand from this example that Bill does not want to go for a run. But that is not what he said. He only said enough for Mary to add the context-mediated information: i.e. someone who is resting doesn’t usually go for a run. The implication is that Bill doesn’t want to go for a run today.

This theory may call to mind the Gricean Maxims — indeed, Sperber and Wilson borrow heavily from Grice’s work.

But I mainly bring up relevance theory to introduce Sperber to those unfamiliar with him. My friend (and Endeca co-founder) Pete Bell recently called to my intention an article by neuroscientist Jonah Lehrer entitled “The Reason We Reason“. The article reviews the “hot hand” fallacy and then proceeds to cite a new theory by Sperber and Hugo Mercier:

Reasoning is generally seen as a mean to improve knowledge and make better decisions. Much evidence, however, shows that reasoning often leads to epistemic distortions and poor decisions. This suggests rethinking the function of reasoning. Our hypothesis is that the function of reasoning is argumentative. It is to devise and evaluate arguments intended to persuade.

The full article by Mercier and Sperber runs over 17K works and is entitled “Why do humans reason? Arguments for an argumentative theory“.

As someone who has spent most of his professional life thinking about information retrieval in practical contexts, I automatically relate relevance theory to relevance in the context of information retrieval. Relevance has been a subject of intense debate in the information science community (Tefko Saracevic tells the story wonderfully). Indeed, a key reason that I created the HCIR workshop was the belief that information retrieval researchers and practitioners (i.e., search engine developers) were placing too much emphasis on an objective notion of topical relevance, and not enough focus on the user.

Mercier and Sperber’s theory offers an interesting challenge to information retrieval researchers: perhaps a user’s information need is less about arriving at the truth and more about finding confirmatory evidence to support a preconceived conclusion. If so, should we adjust our notions of relevance accordingly? Also, if we evaluate or inform search quality based on observed user behavior (such as click-through behavior), then are we already inadvertently conflating topical relevance with users’ confirmatory bias?

Many people have noted that personalization gives us the truth we want: recent examples include Robin Sloan and Matt Thompson’s EPIC 2014 and Eli Pariser’s The Filter Bubble. Despite the consensus that over-fitting information access to our personal tastes is a bad thing (perhaps even dystopian), technology seems to relentlessly push us in this direction. Moreover, some degree of personalization is clearly useful — such as prioritizing information that relates to our personal and professional interests.

Nonetheless, anyone working in the area of information seeking systems should be concerned with the question of the user’s goal in using that system. Many of us take for granted that the user’s main goal is truth seeking, and we design our systems accordingly. What can or should we do differently if the user’s main goal is not informative but persuasive? Is the user looking for an answer…or an argument?

What a day! I’ve been excited about LinkedIn from the moment I joined — and for several years before that — but today has been a unique experience. I hope our celebration extends beyond LinkedIn’s employees and investors — this is a great day for Silicon Valley, for the data scientists who are building its most valuable companies, and for the users who are benefiting from it all. I am proud and deeply grateful to be a part of this extraordinary adventure. My thanks to my hundreds of incredible colleagues and to the 100M users who have made it possible.

ps. Yes, we are still hiring, so please contact me if you’re the kind of person who loves turning data into gold. And if you are local, check out Christos Faloutsos’s upcoming tech talk on Mining Billion Node Graphs, which will take place at LinkedIn on June 2 and is open to the public.

A couple of weeks ago, I participated in a summit that Greylock Partners organized for its portfolio companies at LinkedIn to discuss the power of data. Invited participants represented some of the most interesting “big data” companies in Silicon Valley, including Google, Facebook, Pandora, Cloudera, and Zynga. Discussion took place under the Chatham House Rule, so I’m not at liberty to share much detail. But I can say that there were energetic conversations about metrics, tools, and (of course) hiring.

One of the participants was Google researcher Alon Halevy, who generously shared his presentation on Fusion Tables with me with permission to re-share it here.

Fusion Tables allow the general public to upload, visualize, and share structured data. They are particularly useful for journalists who want to distill compelling stories from data — indeed, The Guardian‘s Simon Rogers has used Fusion Tables to visualize and interpret everything from nuclear power plant accidents to Wikileaks.

After his presentation, I asked Alon for his thoughts on why haven’t we seen an encyclopedic structured data repository comparable in scope, scale to Wikipedia? Alon offered that structured data is brittle — its value tends depend more on context than the unstructured content that populates Wikipedia. I agree in part — for example, consider this map of Brooklyn bus stops that were slated for elimination last summer. Such data is useful in a narrow context, but hardly encyclopedic.

But what about Freebase and DBpedia? Freebase is an open repository of structured data associated with about 20 million topics. DBpedia describes itself as “a community effort to extract structured information from Wikipedia and to make this information available on the Web.” While these tools have seen some use by developers (especially in the semantic web community), they have not achieved mainstream adoption. Perhaps data marketplaces like Factual and Infochimps will be successful as for-profite businesses, but the question remains why we don’t have a Wikipedia-scale success story for public structured data.

I think the problem is easiest to frame in information retrieval terms. Wikipedia is all about precision, but not so much about recall. Let me elaborate.

Wikipedia represents a collective attempt to achieve precision at the level of individual entries. Contributor / editors correct mistakes and argue over the details of content and tone. But coverage is a much lower priority. When in doubt, the Wikipedia collective assumes that information is not notable enough to justify inclusion. Thus Wikipedia errs on the side of precision rather than recall when it comes to meeting the information needs of its users.

This arrangement works well for a typical web user who seeks out information by using Google web search as an interface to discover Wikipedia articles. But structured data is about sets, not just individuals. It does me no good to see aggregate statistics about a set of entities if the set is erratically populated (e.g., Wikipedia’s list of companies established in 1999 or Freebase’s list of those founded after 2000).

In the June 2009 SIGIR Forum, University of Melbourne researchers Justin Zobel, Alistair Moffat, and Laurence Park argued “against recall“, concluding that they could find “no justification for implicit or explicit use of recall as a measure of search satisfaction.” I posted a rebuttal entitled “In Defense of Recall“, arguing that recall is much more useful as a measure for set retrieval than for ranked retrieval. Revisiting this argument two year later, I can see that it holds even more strongly if we are interested in structured data where we want to reason about aggregate properties of sets.

Back when we both worked at Endeca, my colleague Rob Gonzalez described structured data repositories to be as a public good that no one is ever willing to pay for. I’m an optimist by nature, but in this case I fear he has a point. It takes a lot of work to build something useful, and no one seems to have addressed the challenge of incenting people to contribute this work for either economic or altruistic motives.

Or perhaps we’ll just have to wait for the holy grail of information extraction algorithms to structure the world’s information for us? Ironically, that’s not even included on Wikipedia’s list of AI-complete problems.

As regular readers know, I’ve been co-organizing annual workshops on Human-Computer Interaction and Information Retrieval since creating the first HCIR workshop in 2007. These have been a huge success, not only bridging the gap between IR and HCI, but also bringing together researchers and practitioners to address concerns shared by both communities. Past keynote speakers have included such information science luminaries as Susan Dumais, Ben Shneiderman, and Dan Russell.

Every workshop has improved on the previous year’s, and HCIR 2011, which will take place on Thursday, October 20, will be no exception.

Our venue will be Google’s headquarters in Mountain View, California. We could hardly imagine a more appropriate venue: Google has done more than any another company to contribute to everyday information access. Google has been extremely generous as a host and sponsor (other sponsors include Endeca and Microsoft Research), and its location in the heart of Silicon Valley is ideal for attracting researchers and practitioners building the future of HCIR.

Our keynote speaker will be Gary Marchionini, Dean of the School of Information and Library Science at the University of North Carolina at Chapel Hill. Gary coined the phrase ”human–computer information retrieval” in a lecture entitled “Toward Human-Computer Information retrieval“, in which he asserted that ”HCIR aims to empower people to explore large-scale information bases but demands that people also take responsibility for this control by expending cognitive and physical energy.” We are honored to have Gary deliver this year’s keynote.

But of course the main attraction is the contribution of participants. This year we invite three types of papers: position papers, research papers and challenge reports. Possible topics for discussion and presentation at the workshop include, but are not limited to:

• Novel interaction techniques for information retrieval.
• Modeling and evaluation of interactive information retrieval.
• Exploratory search and information discovery.
• Information visualization and visual analytics.
• Applications of HCI techniques to information retrieval needs in specific domains.
• Ethnography and user studies relevant to information retrieval and access.
• Scale and efficiency considerations for interactive information retrieval systems.
• Relevance feedback and active learning approaches for information retrieval.

Demonstrations of systems and prototypes are particularly welcome.

Building on the success of the last year’s HCIR Challenge to address historical exploration of a news archive, this year’s HCIR Challenge will focus on the problem of information availability. The corpus for the Challenge will be the CiteSeer digital library of scientific literature.

For more information about the workshop, including how to submit papers or participate in the challenge, please visit the HCIR 2011 website.

Here are the key dates for submitting position and research papers:

• Submission deadline (position and research papers): July 31
• Notification of acceptance decision: September 8
• Presentations and poster session at workshop: October 20

Key dates for Challenge participants:

• Request access to corpus (contact me) deadline: June 19
• Freeze system and submit brief description: September 25
• Submit videos or screenshots demonstrating systems on example tasks: October 9
• Live demonstrations at workshop: October 20

I’m looking forward to this year’s submissions, and to a great workshop in October. I hope to see many of you there!

As I posted a few months ago, I’m organizing the Industry Event at CIKM 2011 with Tony Russell-Rose. We have a great set of keynotes lined up:

• Stephen Robertson (Microsoft Research)
• John Giannandrea (Google)
• Jeff Hammerbacher (Cloudera)
• Peter Jackson (Thomson Reuters)

We’re also looking for submissions from industry researchers and practitioners. The submission deadline is June 21.

Here is a copy of the call for papers:

This year’s CIKM conference will include an Industry Event, which will be held during the regular conference program in parallel with the technical tracks.

The Industry Event’s objectives are twofold. The first objective is to present the state-of-the-art in information retrieval, knowledge management, databases, and data mining, delivered as keynote talks by influential technical leaders who work in industry. The second objective is to present interesting, novel and innovative industry developments in these areas.

Industry authors are invited to prepare proposals for presenting interesting, novel and innovative ideas, and submit these to industry@cikm2011.org by June 21st 2011. The proposals should contain (with respective lengths):

• Short company portrait (125 words)
• Short CV of the presenter (125 words)
• Title and abstract of the presentation (250 words)
• Reasons why the presentation should be interesting to the CIKM audience

When submitting a proposal, please bear in mind the following:

• Ensure the presentation is relevant to the CIKM audience (the Call for Papers gives a good idea of the conference scope).
• Try to highlight interesting R&D challenges in the work you present. Please do not present a sales pitch.
• All slides will be made public (no confidential information on the slides; you will be expected to ensure your slides are approved by your company before being presented).
• Presenters may opt to have their presentation videoed and made public, and if so, the presenter will be asked to sign a release form.

We look forward to receiving your submissions, and welcoming you to the CIKM 2011 Conference and Industry Event.

Important dates:
21 June 2011: Industry Event paper proposals due
19 July 2011: Notifications sent
27 October 2011: Industry Event
24-28 October 2011: CIKM conference

Pankaj Mehra and I are guest editors for an upcoming special issue of IEEE Internet Computing with the topic “Beyond Search: Context-Aware Computing“.

Here is a copy of the call for papers:

Context is the unstated actor in human communications, actions, and situations. It makes our communication efficient, our commands actionable, and our situations understandable to the people, organizations, and devices that provide us with content or services. The increased embedding of technology into our personal and social environments drives a need for context-aware computing.

Context-aware computing offers mobile Internet users an experience that goes beyond user-initiated search and location-­based services. Context awareness sharpens relevance when responding to user-initiated actions (such as product search and support calls). It also enables proactive communications through analysis of a user’s behavior and environment, thereby forming the basis for key business imperatives targeting customer-engagement systems. Even greater opportunity arises from context use in systems that can make sense of and engage in customer dialogs and forums.

This special issue seeks original articles that support and illustrate context use in creating enhanced user experiences. Sample topics include

• proactive, contextualized delivery of information, alerts, and advertisements;
• context-mediated Web service orchestration, yielding actionable interpretation of spoken high-level commands;
• system architecture, economics, and ecosystems for comprehensive capture, representation, communication, gathering, and brokering the larger user context;
• systems of engagement that treat discourse as text plus context and process textual communication as an event in which linguistic, cognitive, and social actions converge; and
• reasoning and knowledge representation mechanisms that use context in selecting the body of knowledge to use, the level of detail to model, and the point of view with which to communicate and interpret text and data.

All submissions must be original manuscripts of fewer than 5,000 words, focused on Internet technologies and implementations. All manuscripts are subject to peer review on both technical merit and relevance to IC’s international readership—primarily system and software design engineers. We do not accept white papers, and we discourage strictly theoretical or mathematical papers. To submit a manuscript, please log on to ScholarOne (https://mc.manuscriptcentral.com:443/ic-cs) to create or access an account, which you can use to log on to IC’s Author Center and upload your submission.

I hope some of you will submit articles in time for the June 15 deadline, and Pankaj and I look forward to reviewing them.

I thought I’d prepare for the discussion by sharing my thoughts here. Perhaps some of you will even be kind enough to add your own ideas, which I promise to share with the reading group.

I encourage you to read the paper, but here’s a summary of its results:

• A user’s influence on Twitter is the extent to which that user can cause diffusion a posted URL, as measured by reposts propagated through follower edges in Twitter’s directed social graph.
• The best predictors of future total influence are follower count and past local influence, where local influence refers to the average number of reposts by that user’s immediate followers, and total influence refers to average total cascade size.
• The content features of individual posts do not have identifiable predictive value.
• Barring a high per-influencer acquisition cost, the most cost-effective strategy for buying influence is to target users of average influence.

Let’s dive in a bit deeper.

The definitions of influence and influencers are, by the authors’ own admission, narrow and arbitrary. There are many ways one could define influence, even within the context of Twitter use. But I agree with the authors that these definitions have enough verisimilitude to be useful, and their simplicity facilitates quantitative analysis.

It’s hardly surprising that past influence is a strong predictor of future influence. But it might seem counterintuitive that, for predicting future total influence, past local influence is more informative than past total influence. The authors suggest the explanation that most non-trivial cascades are of depth 1 — i.e., total influence is mostly local influence. But at most that would make the two features equally informative, and total influence should still be a mildly better predictor.

I suspect that another factor is in play — namely, that the difference between local influence and total influence reflects the unpredictable and rare virality of the content (e.g., a random Facebook Question generated 4M votes). If this hypothesis is correct, then past local influence factors out this unpredictable factor and is thus a better predictor of both future local influence and future total influence.

I’m a bit surprised that follower count supplies additional informative value beyond the past local influence; after all, local influence should already reflect the extent to which the followers are being influenced. It’s possible that past influence lags the follower count, since it does not sufficiently weigh the potential contributions of more recent followers. But another possibility is one analogous to the predictive value of past local vs. global influence: past local influence may include an unpredictable content factor which follower count factors out.

Of course, I can’t help suggesting that TunkRank might be a more useful indicator than follower count. Unfortunately the authors don’t seem to be aware of the TunkRank work — or perhaps they preferred to restrict their attention to basic features.

I’m not surprised by the inability to exploit content features to predict influence. If it were easy to generate viral content, everyone would do it. Granted, a deeper analysis might squeeze out a few features (like those suggested in the Buddy Media report), but I don’t think there are any silver bullets here.

Finally, the authors consider the question of designing a cost-effective strategy to buy influence. The authors assume that the cost of buying influence can be modeled in terms of two parameters: a per-influencer acquisition cost (which is the same for each influencer) and a per-follower cost for each influencer. They conclude that, until the acquisition cost is extremely high (i.e., over 10,000 times the per-follower cost), the most cost-efficient influencers are those of average influence. In other words, there’s no reason to target the small number of highly influential users.

The authors may be arriving at the right conclusion (Watts’s earlier work with Peter Dodds, which the paper cites, questions the “influentials” hypothesis), but I’m not convinced by their economic model of an influence market. It may be the case that professional influencers are trying to peddle their followers’ attention on a per-follower basis — there are sites that offer this model.

But why should anyone believe that an influencer’s value is proportional to his or her number of followers? The authors’ own work suggests that past local influence is a more valuable predictor than follower count, and again they might want to look at TunkRank.

Regardless, I’m not surprised that a fixed per-follower cost makes users with high follower counts less cost-effective, as I subscribe to its corollary: as a user’s follower count goes up, the per-follower value diminishes. I haven’t done the analysis, but I believe that the ratio of a user’s TunkRank to the user’s follower count tends to go down as a user’s follower count goes up. A more interesting research (and practical) question would be to establish a correctly calibrated model of influencer value and then explore portfolio strategies.

In any case, it’s an interesting paper, and I look forward to discussing it with my colleagues next week. Of course, I’m happy to discuss it here in the meantime. If you’re in my reading group, feel free to chime in. And you’re not in you’re not in my reading group, consider joining. We do have openings.

I like Google…

I’ve been a regular Google user since the day I first discovered its existence in 1999. Indeed, I’ve consistently found Google to be the most useful service on the web. That’s not love, but it’s a very strong +1.

Moreover, I’d say that my preference for Google is an informed one. I’ve given all of the major search engines a fair chance, and even tried a fair number of obscure ones. They all have their strengths, but none have delivered enough utility to me to justify the cognitive load of using more than one search engine for the open web.

…but I don’t need Google.

Nonetheless, I know that, if Google disappeared tomorrow or became inconvenient to access, I’d be content with one of its competitors. I have no particular investment in Google beyond brand loyalty.

Actually, that’s not entirely true. I could easily walk away from Google search, but I’d be apoplectic if I suddenly lost access to my Gmail account — much as if I lost access to my LinkedIn or Twitter accounts. Indeed, Gmail is the only way in which Google has me locked in, but I don’t see my Gmail account as entangled with my access to Google’s other services.

Perhaps that not a bug but a feature: after all, Google trumpets the virtues of “open” and the portability of user data (including Gmail) through the Data Liberation Front. Nonetheless, it’s no secret that Google has a major case of Facebook envy. And if rumors hold, Google is now making the success of its social strategy a major component in all employee compensation.

Social is Give to Get.

Google critics often assert that Google doesn’t get social. But I think the problem isn’t so much with what Google gets as what it gives. When it comes to social, you have to give to get. That is, to get data and engagement, you have to provide social utility.

To start off, Google would love to know who you are. That’s why it developed Google Profiles in 2007. People are more than willing to provide data about who they are, as proven by the hundreds of millions of people who create profiles on Facebook and LinkedIn. Perhaps Google was a little bit late to the game. More likely, people didn’t see enough utility in creating Google profiles. Facebook, on the other hand, helps people be found by their friends and family in a context designed for social interaction. LinkedIn offers people the opportunity to be found by people who can help you professionally: colleagues, classmates, potential employers, etc. Google didn’t give people much reason to invest effort — in fact it seems to treat Profiles as a dumping ground populated by Google’s other products, rather than valuable piece of online real estate embedded in a living social context. Not surprisingly, users invest their efforts elsewhere.

Google would also love to know where you are and where you’ve been — that’s why Google created Latitude in 2009. Moreover, Google developed this pioneering location-based service as a complement to Google Maps, perhaps the best product Google has produced outside of search. Given it’s dominance in mapping services, directions, and local search, Google should be the leader of all things local. And yet, while Latitude has flopped, Foursquare — which launched in the same year as a tiny startup after Google acquired and shut down its previous incarnation– succeeded in defining location-based services as a category. Before Foursquare, the idea of a service tracking your location was one that most of us associated with Lo-Jack and Big Brother — if not with modern totalitarian regimes. Yet, by making a game out of “checking in” to venues, Foursquare inspired its users to willingly — and eagerly! — share and publish their whereabouts. It’s unclear whether this model will create sustained interest (cf. Mark Watkins’s analysis at ReadWriteWeb), but Foursquare’s success thus far is predicate on its offers social utility in exchange for data and attention.

Of course, Google also wants to know what you like. That’s why Google developed SearchWiki (RIP), Hotpot (now merged into Places), and most recently +1. As Amazon, Facebook, Netflix, and Yelp have demonstrated, people aren’t shy about sharing their opinions publicly, given the right social context and utility. Unfortunately, Google seems to struggle with that last part. Google embedded SearchWiki in the non-social context of search — and has launched +1 the same way. It’s not at all clear what users would gain by going out of their flow to annotate search results. Hotpot may simply be a case of too little, too late — people are already trained to go to Yelp and Facebook Fan pages for subjective information about service businesses. Overall, Google has not given users a reason to believe there is significant return on their investment in sharing opinions.

Collecting Data Doesn’t Count.

Of course Google is able to collect a significant amount of data about users’ identities through their search history, cookies, browser toolbars, and purchase history (if they use Google Checkout). Indeed, it is Google inference of user intent in search queries that has allowed Google to become the poster child of online advertising.

But collecting data is not the same as having the user volunteer it. Most users have a transactional relationship with Google, tolerating data collection and advertising in exchange for a free service. Google wants more — it wants users to invest in identities associated with their Google accounts. But Google doesn’t seem to undertand that users don’t make these investments unless their receive some social or professional utility in return.

If it’s true that Larry Page is making “social” Google’s top OKR, then I hope for the sake of my former colleagues that Google has learned from its past experiments.

Note: This post was written by Scott Nicholson, a Senior Data Scientist at LinkedIn. Scott is data and modeling geek with a passion for startups, product and user experience. His work at LinkedIn focuses on analyzing and improving user engagement and monetization.

I’m happy to report back on my experience at the Data 2.0 conference, an event organized by midVentures and targeted at entrepreneurs building products to leverage the dramatic increase in publicly and privately collected data. The conference has four main themes: what data is available, how to obtain data, how to store and access data, and how to create value from data products. For data nerds or hackers, the conference offered a delightful stream of  “you know what would be cool…” ideas.

The morning started off on a strong foot with a talk by Vivek Wadhwa on how data is going to define the next generation of successful startups in a new information age. He observed the increasing online access to data that has previously been restricted to offline access (or no access at all). He also emphasized the importance of  new sources of data, such as medical records and genome data. We need to think of social use of data beyond Twitter, Facebook and LinkedIn: for example, genome data will allow us to connect to each other in ways that helps us better understand our similarities and differences. Meanwhile, some existing data sources will become increasingly open and available to all. Wadhwa stressed the importance of leveraging the open sources of federal, state and local government data to come up with solutions to the existing closed and clunky legacy systems that governments used to generate data reports (a pity that data.gov and related programs may be defunded — DT).

The morning keynote segued nicely into the panel on open data sources. Jay Nath, Director of CRM for the city of San Francisco, noted that, while many applications are using government data and APIs, they mostly address consumer convenience (e.g., public transit apps) rather than government efficiency.  Panelists agreed that government employees have few incentives to take risks by using new technology: legacy systems might be expensive, inflexible and inefficient, but they do perform their limited function. Alluding to Eric Ries’s idea of a “lean startup“, Nath suggested the concept of a “lean government” that lowered costs, sped up its operations, and avoided procurement processes by using open source technology — all in the context of providing services to its citizens.

The inspiring mid-day keynote by former Amazon Chief Scientist Andreas Weigend took a different perspective from the morning sessions: he focused on the how data sharing can provide tangible value to end-users, even resulting in significant behavior change. He cited products like tweeting weight scales, FitBit, and Nike + that allow people to share data about their fitness efforts, thus leading to social reinforcement for positive behaviors. I personally see this area as a great example of where data scientists and engineers can create enormous economic value and increase people’s welfare

The day also featured a various product launches and presentations. Here are a few that caught my attention:

• Micello: Google maps for indoors. They won the startup competition that was held in conjunction with the conference.
• Tropo: API for voice calls and SMS
• DataStax Brisk: Technology unifying Hadoop, Hive & Cassandra. A new Hadoop distribution powered by Cassandra.
• Neer: always-on location awareness app from Qualcomm. Privately share location with groups and families.
• Heritage Health Prize: $3MM prize for predictive modeling around who will require hospitalization (a follow-up on their announcing the prize at Strata)

Overall, it was great to see hundreds of people exploring innovations and opportunities to use data to improve business, technology and society.

My ideas are somewhat predictable, in that they all address user-centric information-seeking problems. Working for over a decade in this space has focused my intellectual curiosity somewhat — and of course I work on a number of these problems at LinkedIn. But there are many information-seeking problems that are outside of my present or foreseeable scope.

Here are two ideas that I’m hoping someone will execute on so I don’t have to:

1. Shopping: Help Me Figure Out What I Want

We’ve come a long way to improve the shopping experience, at least for utilitarian shoppers like yours truly. If I know exactly what I want, I usually find it by using Google as a gateway to Amazon, taking a bit more time if I’m feeling price-sensitive. I’d happily install a browser extension that could automatically detect product search queries and take them to my preferred shopping sites, bypassing the search results page, but that’s a minor detail of convenience (though probably not such a minor detail for the search engine companies). In any case, known-item search for online shopping is hardly inspiring as an open problem.

Exploratory search is another story entirely. For all the work that’s been done on faceted search, it is used almost exclusively to help people narrow search results. Progressive narrowing is great if you have a pre-established information need, but it is not the best interface if you’re hoping to evolve your information need through exploration. Instead of just “help me find what I’m looking for”, I’d also like to see more “help me figure out what I want”. I’d like to see an innovator applying faceted search to broaden queries, not just to narrow them, as well as going beyond collaborative filtering and “related items” to create a compelling browsing experience based on semantic and social navigation.

2. Organizing the World’s Information: Beyond Wikipedia and Navigational Queries

If shopping online often reduces to using Google to find product pages on Amazon, then informational queries similarly reduce to using Google to find Wikipedia entries. Nothing against Wikipedia — I think it is one of the most extraordinary achievements of our generation — but I think of the web as a library and Wikipedia as its encyclopedia section. Google’s mission statement notwithstanding, web search engines do a poor job of organizing the rest of the world’s information, instead choosing to optimize for known-item search.

There are countless opportunities for improvement here. Imagine if there were an interface for books, scholarly articles, patents, music, or videos that supported browsing and exploration of their content and meta-data. We’ve seen the beginnings of such an approach for individual libraries (e.g., the Triangle Research Libraries Network), but there is so much more to do in this space. Perhaps it’s a space that is hard to monetize, but even then I’d expect philanthropists to take an interest in making the world’s knowledge and creative artifacts more accessible.

If you are pursuing either of these areas, I’d love to hear about it. I’m sure readers here would too. I’m also curious to learn more about innovation in the travel and personals spaces, as those are both areas that could benefit from supporting exploratory search. And if you have work in progress, please present it at the HCIR workshop!

This afternoon, I met with a couple of Stanford seniors to advise them on a startup they’ve been developing and targeting towards mid-sized online retailers. I’d expected to spend most of the time talking about their technology and customer development strategy — and we did indeed talk about these things. But we spent most of the time brainstorming whom I knew that could best help them achieve the key milestone of landing a first customer.

Not surprisingly, my first step was to open up my laptop and head straight to LinkedIn (I’m not only a data scientist — I’m also a member!) to see who in my network might be most helpful to them at this critical stage. The students were openly impressed: despite being sharp, energetic, and remarkably business-savvy for a couple of guys not old enough to legally buy beer, they had never seen someone use LinkedIn the way I was doing in front of them — not for hiring or recruiting, but as an exploratory search tool to find useful professional connections.

I started with a search for online retail, then restricted to directors and VPs, narrowing down further to first-degree and second-degree connections. I vetted second-degree connections by looking at my paths to them, determining who would be likely to be most helpful either because they owed me a favor or because they might have their own interest in the startup’s success.

We then browsed through the list of top online retailers, identifying plausible companies for them to target and then looking for my first-degree and second-degree connections not only at those companies but also at other companies in the same space. We spent over an hour fluidly going back and forth between talking and exploring on LinkedIn. In the course of this exploration, we not only produced a list of people to contact, but also arrived at a better understanding of the business strategy.

I’m always happy to help young entrepreneurs who represent the future of our economy, and even happier to do so using the tools my colleagues and I are constantly working to improve. But I’m surprised and a bit disheartened that the methods I used are not common knowledge, especially among people who stand to gain the most benefit from them. Perhaps, as someone who has been using LinkedIn since 2004, I take for granted that people know how to take advantage of it for professional networking. I hope that the company’s increasing visibility will make more people aware that LinkedIn is not *just* the best things that has ever happened to recruiting.

Also, as an HCIR advocate, I’d like to see these kinds of information-seeking tasks receive more attention from researchers and practitioners. I’ve been saying for a while that these and similar tasks that are neglected by the information retrieval community and not adequately addressed by Google . For example, while there has been significant research effort in the area of expert finding, I’d like to see more efforts to improving the interactive process of finding experts and expertise. And not just from LinkedIn!

If you are doing work in this space, I hope you’ll participate in the upcoming HCIR workshop and show off your stuff.

In the meantime, I hope you make the most of LinkedIn, for fun and for profit. As a mentor of mine told me in my first job, it’s “network or not work”.

My feelings about software patents are a matter of public record (e.g., this open letter to the USPTO). As things stand today, software patents act as an innovation tax rather than as a catalyst for innovation. It may be possible to resolve the problems of software patents through aggressive reform, but it would be better to abolish software patents than to maintain the status quo.

My personal feelings notwithstanding, I acknowledge the reality that today’s software companies need to have defensive patent strategies. In a previous job, one of my key accomplishments was to hire a director of intellectual property. It was a difficult hire, but it happened just in time to defend against a particularly noxious patent troll. I am not at liberty to spell out the details, but I can say that we responded with a long, expensive fight that effectively quashed the patent and the lawsuit.

Beware Of Trolls

Patent trolls, known less pejoratively as non-practicing entities (NPE) because they do not actually sell products or services that implement the systems or methods in the patents they own, take advantage of asymmetric risk. On one hand, an NPE does not need much money to bankroll (or at least initiate) a patent infringement suit — in fact, there are law firms who will take such cases on contingency. On the other hand, the company being sued faces potentially ruinous costs. Moreover, even if a company feels certain that a lawsuit against it is baseless, the company cannot count on the imperfect and inefficient legal system to reach a fair outcome. As a result, the company has to choose between spending heavily in its own defense or settling with the NPE. Most companies opt for the less risky route and negotiate settlements, providing funds that the NPEs use to sue more companies.

Some people have a name for this style of asymmetric warfare — namely, terrorism. I suppose that the word terrorist is loaded enough without increasing its breadth to include patent trolls — not to mention that trolls have their defenders. But the metaphor is a useful one. A terrorist attack inflicts an amount of damage that is much greater than the absolute cost to the terrorist, e.g., a suicide bomber who inflicts mass murder. Moreover, the threat of terrorism puts the object of that threat in the position between settling (aka negotiating with terrorists) or spending heavily on counter-terrorism efforts. As Peter Neumann notes in a Foreign Affairs article:

The argument against negotiating with terrorists is simple: Democracies must never give in to violence, and terrorists must never be rewarded for using it. Negotiations give legitimacy to terrorists and their methods…

Yet in practice, democratic governments often negotiate with terrorists.

There have been various attempts to address the threat of patent trolls.

Google litigation director Catherine Lacavera has gone on record saying that Google intends to fight rather than settle patent infringement lawsuits in order to deter patent trolls. We’ll see if Google can sustain this “we don’t negotiate with terrorists” approach; I admire the resolve, but like Neumann I’m skeptical.

Article One Partners has built a business around crowd-sourcing patent invalidation. Clients pay for research to invalidate patents, and Article One offers bounties to anyone who contributes valuable evidence. In theory, companies can request validity analysis of their own patents to test them for robustness, but I assume that the primary application of this service is the invalidation patents that a company sees as threats.

Rational Patent (RPX) has created a defensive patent pool. purchasing a large portfolio of patents and then licensing them to its member companies. Some have questioned whether this approach is “patent extortion by another name“, and indeed paying RPX for a blanket license does feel a bit like preemptively settling in bulk. But I’d be more concerned that the “over 1,500 US and international patent assets” that RPX claims to have acquired are a drop in the bucket compared to the vast number of patents that the USPTO has granted, many of dubious merit.

Meanwhile, patent trolldom is serious business. Former Microsoft CTO Nathan Myhrvold created Intellectual Ventures to “invest both expertise and capital in the development and monetization of inventions and patent portfolios.” The company has only filed one lawsuit so far, but Mike Masnick claims that it has used over a thousand shell companies to conduct stealth lawsuits.

Unfortunately, the proliferation of lawsuits by software patent trolls suggests that the economic incentives encourage such suits. If every company could sustain a “Never give up, never surrender!” approach, patent trolls would eventually go away, but it is unlikely that companies would be willing to assume the short-term risks that such an approach entails.

Moreover, this approach only works if everyone participates, requiring that every company forgo the competitive advantage it could enjoy from being the only company among its competitors to appease the trolls. This is a classic tragedy of the commons. I’m hopeful that we’ll eventually implement sensible patent reform in the United States, but I expect it will take a long time to overcome the entrenched interests that support the status quo.

It’s Not Just The Trolls

But NPEs are not the only cause for concern. Many established companies, including some technology leaders, are not averse to using patent lawsuits as part of their business strategy. The mobile device and software space is a particularly popular arena for patent litigation, the most notable being Oracle’s lawsuit against Google claiming that Android infringes on patents related to Java. The stakes are extraordinary, dwarfing even the $612.5M that RIM paid NTP in order to avoid a complete shutdown of the BlackBerry service (ironically, at least some of the patents involved have since been rejected by the patent office after re-examination).

Patent lawsuits can also be a way for larger companies to bully smaller ones. For example, a couple of entrepreneurs at visual search startup Modista were forced to shut down their company because of a lawsuit by Like.com, a more established player in the space which was ultimately acquired by Google. Note: although I was Google at the time, I have no inside knowledge of the acquisition, nor whether there is any truth to the speculation that Google acquired the company for its patents.

Defensive Patenting

Moral considerations aside, the above stories make it clear that defensive patent strategy isn’t just about NPEs. In fact, many software companies take an approach to defensive patenting is to assemble a trove of patents that are useful for countersuits and thus serve as a deterrent. Back to military metaphors, it’s similar to countries developing nuclear weapons (a popular metaphor for patents in general) in accordance with the doctrine of mutual assured destruction.

Companies that follow a defensive patent strategy typically implement a process for capturing intellectual property. Scientists and engineers file invention disclosures, a committee reviews these for patentability, and a law firm translates the invention disclosures into patent filings. The filings then go through the meat grinder of patent prosecution and eventually are extruded as patents.

It all sounds great in theory — indeed, I have seen executives who mostly worry about educating scientists and engineers about patents and providing the right incentives to encourage them to write and submit invention disclosures. Indeed, it can be difficult to integrate intellectual property capture into the process and culture of a software company. But I think there are two much bigger issues.

First, it takes several years to obtain a patent. Indeed, the USPTO dashboard shows that it takes two years just to get an initial response from the patent office. Thus a defensive patenting strategy requires significant advance planning: any patents filed today are unlikely to be useful deterrents until at least 2014. Given the rapid pace of the software industry, this delay is very significant. Moreover, startups are especially vulnerable in their first few years.

Second, intellectual property capture processes are inherently optimized for offensive (i.e., don’t copy my invention or I’ll sue you) rather than defensive (i.e., don’t sue me) patent strategy. Consider Google’s defensive position with respect to Oracle in the aforementioned lawsuit. Google has a relatively small patent portfolio, but it has obtained patents for some of its major innovations, such as MapReduce. Let’s put aside questions about the validity of the MapReduce patent — especially since patents enjoy the presumption of validity. The bigger question is to whom such a patent serves as a deterrent against patent lawsuits. It may very well deter Hadoop users, which include Google arch-rival Facebook. But, as far as I know, Oracle is not vulnerable on this front. FOSS Patents blogger Florian Mueller did an analysis and concluded that Google’s patents are not an effective deterrent. Indeed, the fact that Google has not counter-sued Oracle using its own patents is at least consistent with this analysis.

What if Google were to invest in obtaining (i.e., purchasing) a collection of broad patents that had to do with relational databases? Such patents could have nothing to do with Google’s areas of innovation and nonetheless serve as an effective deterrent against lawsuits from relational database companies like Oracle. Even if the patents were not robust, they would still have some value as deterrents because of their presumption of validity and the aforementioned inefficiency of the legal system.

In general, the most valuable defensive patents are those that you believe your competitors (or anyone else who might have an interest in suing you) are already infringing. Even if those patents would be unlikely to survive re-examination, the re-examination process is long and expensive, and even the most outrageous of patents enjoys the presumption of validity.

Everybody Into The Pool

A patent pool is a consortium of at companies that agree to cross-license each other’s patents — a sort of mutual non-aggression pact. But perhaps companies that only believe in the defensive use of patents should take a more aggressive approach to patent pooling. Following the example of NATO, they could create an alliance in which they agree to mutual defense in response to an attack by any external party. I don’t know if such an approach would be viewed as anti-competitive, but it does strike me as a cost-effective alternative to the current approach for defensive patenting.

And, as with most ideas, this one is hardly original. In 1993, Autodesk founder John Walker published “PATO: Collective Security In the Age of Software Patents“, in which he proposed:

The basic principle of NATO is that an attack on any member is considered an attack on all members. In PATO it works like this–if any member of PATO is alleged with infringement of a software patent by a non-member, then that member may counter-sue the attacker based on infringement of any patent in the PATO cross-licensing pool, regardless of what member contributed it. Once a load of companies and patents are in the pool, this will be a deterrent equivalent to a couple thousand MIRVs in silos–odds are that any potential plaintiff will be more vulnerable to 10 or 20 PATO patents than the PATO member is to one patent from the aggressor. Perhaps the suit will just be dropped and the bad guy will decide to join PATO….

Since PATO is chartered to promote the free exchange and licensing of software patents, members do not seek revenue from their software patents–only mutual security. Thus, anybody can join PATO, even individual programmers who do not have a patent to contribute to the pool–they need only pay the nominal yearly dues and adhere to the treaty–that any software patents they are granted will go in the pool and that they will not sue any other PATO member for infringement of a software patent.

It’s been almost two decades, but perhaps PATO is an idea whose time has come. And, even if a collective effort fails, individual companies might do well to focus less on intellectual property capture and more on collecting the kinds of nuisance patents currently favored by trolls. After all, the best defense is the credible threat of a good offense.

Conclusion

Even if you hate software patents, you can’t afford to ignore them if you are in the software industry. And I’m well aware that not everyone shares my view of software patents. But I hope those who do find useful advice in the above discussion. I’d love to see the software industry move beyond this innovation tax.

A couple of years ago, I wrote a post entitled “Life, the Universe, and SEO” in which I considered Google’s relationship with the search engine optimization (SEO) industry. Specifically, I compared it to the relationship that Deep Thought, the computer in Douglas Adams’s Hitchhikers Guide to the Galaxy, has with the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons.

Interestingly, both SEO and union protests have been front-page news of late. I’ll focus on the former.

Three recent incidents brought mainstream attention to the SEO industry:

• Two weeks ago, Google head of web spam Matt Cutts told the New York Times that Google was engaging in a “corrective action” that penalized retailer J. C. Penney’s search results because the company had engaged in SEO practices that violated Google’s guidelines. For months before the action (which included the holiday season), J. C. Penney was performing exceptionally well in broad collection of Google searches, including such queries as [dresses], [bedding], [area rugs], [skinny jeans], [home decor], [comforter sets], [furniture], [tablecloths], and [grommet top curtains]. As I write this blog post, I do not see results from jcpenney.com on the first result page for any of these search queries.
• This past Thursday, online retailer Overstock.com reported to the Wall Street Journal that Google was penalizing them because of Overstock’s now discontinued practice of rewarding students and faculties with discounts in exchange for linking to Overstock pages from their university web pages. Before the penalty, these links were helping Overstock show up at the top of result sets for queries like [bunk beds] and [gift baskets]. As I write this blog post, I do not see results from overstock.com on the first result page for either of these search queries.
• That same day, Google announced, via an official blog post by Amit Singhal (Google’s head of core ranking) and Matt Cutts, a change that, according to their analysis, noticeably impacts 11.8% of of Google search queries. In their words: “This update is designed to reduce rankings for low-quality sites—sites which are low-value add for users, copy content from other websites or sites that are just not very useful. At the same time, it will provide better rankings for high-quality sites—sites with original content and information such as research, in-depth reports, thoughtful analysis and so on.”

Of course, Google is always working to improve search quality and stay at least one step ahead of those who attempt to reverse-engineer and game its ranking of results. But it’s quite unusual to see so much public discussion of ranking changes in such a short time period.

Granted, there is a growing chorus in the blogosphere bemoaning the decline of Google’s search quality. Much of it focused on “content farms” that seem to be the target of Google’s latest update. Perhaps Google’s new public assertiveness is a reaction to what it sees as unfair press. Indeed, Google’s recent public spat with Bing would be consistent with a more assertive PR stance.

But what I find most encouraging is the Google’s recent release of Chrome browser extension that allows users to create personal site blocklists that are reported to Google. Some may see this is as a reincarnation of SearchWiki, an ill-conceived and short-lived feature that allowed searchers to annotate and re-order results. But filtering out entire sites for all searches offers users a much greater return on investment than demoting individual results for specific searches.

Of course, I’d love to see user control taken much further. And I wonder if efforts like personal blocklists are the beginning of Amit offering me a more positive answer to the question I asked him back in 2008 about relevance approaches that relied on transparent design rather than obscurity.

I’m a realist: I recognize that many site owners are competing for users’ attention, that most users are lazy, and that Google wants to optimize search quality subject to these constraints. I also don’t think that anyone today threatens Google with the promise of better search quality (and yes, I’ve tried Blekko).

Perhaps the day is in sight when human-computer information retrieval (HCIR) offers a better alternative to the organization of web search results than the black-box ranking that fuels the SEO industry. But I’ve waiting for that long enough to not be holding my breath. Instead, I’m encouraged to see a growing recognition that today’s approaches are an endless game of Whac-A-Mole, and I’m delighted  that at least one of the improvements on the table takes a realistic approach to putting more power in the hands of users.

Heading to Punta Cana for a week. Feel free to keep writing great comments — will catch up when I get back!

Unless you’ve been living in a cone of silence, you’ve probably heard about the epic war of words between Google and Bing. But just in case, here’s a quick summary:

Amit Singhal, Google Fellow: “Microsoft’s Bing uses Google search results—and denies it“:

Bing is using some combination of:

• Internet Explorer 8, which can send data to Microsoft via its Suggested Sites feature
• the Bing Toolbar, which can send data via Microsoft’s Customer Experience Improvement Program

or possibly some other means to send data to Bing on what people search for on Google and the Google search results they click. Those results from Google are then more likely to show up on Bing. Put another way, some Bing results increasingly look like an incomplete, stale version of Google results—a cheap imitation.

Harry Shum, Corporate Vice President, Bing: “Thoughts on search quality“:

We use over 1,000 different signals and features in our ranking algorithm. A small piece of that is clickstream data we get from some of our customers, who opt-in to sharing anonymous data as they navigate the web in order to help us improve the experience for all users.

Yusuf Mehdi, Senior Vice President, Online Services Division, Bing: “Setting the record straight“:

Google engaged in a “honeypot” attack to trick Bing. In simple terms, Google’s “experiment” was rigged to manipulate Bing search results through a type of attack also known as “click fraud.” That’s right, the same type of attack employed by spammers on the web to trick consumers and produce bogus search results. What does all this cloak and dagger click fraud prove? Nothing anyone in the industry doesn’t already know. As we have said before and again in this post, we use click stream optionally provided by consumers in an anonymous fashion as one of 1,000 signals to try and determine whether a site might make sense to be in our index.

Matt Cutts, Head of Webspam, Google: “My thoughts on this week’s debate“:

Something I’ve heard smart people say is that this could be due to generalized clickstream processing rather than code that targets Google specifically. I’d love if Microsoft would clarify that, but at least one example has surfaced in which Microsoft was targeting Google’s urls specifically. The paper is titled Learning Phrase-Based Spelling Error Models from Clickthrough Data and here’s some of the relevant parts:

The clickthrough data of the second type consists of a set of query reformulation sessions extracted from 3 months of log files from a commercial Web browser [I assume this is Internet Explorer. --Matt] …. In our experiments, we “reverse-engineer” the parameters from the URLs of these [query formulation] sessions, and deduce how each search engine encodes both a query and the fact that a user arrived at a URL by clicking on the spelling suggestion of the query – an important indication that the spelling suggestion is desired. From these three months of query reformulation sessions, we extracted about 3 million query-correction pairs.”

This paper very much sounds like Microsoft reverse engineered which specific url parameters on Google corresponded to a spelling correction. Figure 1 of that paper looks like Microsoft used specific Google url parameters such as “&spell=1″ to extract spell corrections from Google. Targeting Google deliberately is quite different than using lots of clicks from different places.

Let me start by saying that these are very serious words from very serious people.

Amit and Matt, both of whom I know personally, are not just two of the most prominent Google employees — they have a deep personal investment in Google’s search quality. Amit is personally responsible for much of Google’s web search ranking algorithm, and Matt is surely the person whom spammers (and many SEO consultants) most love to hate. There is no question in my mind that the emotion both of them are expressing is sincere.

I haven’t met Harry or Yusuf, but I have no reason to doubt their own sincerity — especially since everything they are saying seems consistent with the facts — in fact, consistent with the substantive parts of Google’s allegations. Indeed, the facts don’t really seem to be in dispute. And more generally, I’ve met some of the folks who lead the Bing team (like Jan Pedersen), and, like Matt, I believe they are thoughtful and sincere and are devoted to building a great search engine of their own.

The debate is not about the facts. Rather, it’s about what is right and wrong. I will try to summarize the two sides’ position without editorializing.

Bing is claiming that:

• Users have a right to do as they please with their own clickthrough data, which includes data from Google search sessions.
• Bing toolbar users opted in to share this clickthrough data with Bing.
• By using this clickthrough data, Bing creates value for users.

Google is claiming that:

• Bing’s specific targeting of Google clickthrough data amounts to copying Google and is wrong.
• Bing toolbar users are not necessarily aware that they are complicit in this behavior.
• Bing is disingenuous in understating how much it benefits from Google as a signal.

What do I think?

I agree with Bing that users have the right to do as they please with clickthrough data. I’d think Google would agree too, given that Google wrote the sermon on “the meaning of open“:

Open information means that when we have information about users we use it to provide something that is valuable to them, we are transparent about what information we have about them, and we give them ultimate control over their information.

I agree with all of the three points I listed as Google’s claims except for the part that Bing’s behavior is wrong. It’s up to users if they want to help Bing compete with Google. Do users know that they’re doing so? Probably not. But would they stop doing so if they did? I doubt it. I can’t see why most users would have a dog in this fight — and in fact, it may be in users’ interest to help Bing be more competitive.

I do think Bing should be forthright about what it is doing — and how much this user-provided data from Google search sessions is contributing to its own quality improvements. Bing can, of course, keep this information secret, but I’d think that Bing would want to defend its reputation as an innovator — especially as the David in a David vs. Goliath fight.

But I also think that Google should be careful with its accusations. Accusing Bing of not being innovative is one thing, and that accusation, backed by concrete examples, is probably enough to score points. But implying that Google owns its users’ clickthrough data and that Bing has no right to solicit that data from users is another thing entirely.

I’m curious to hear what others here think. It’s been a while since I could freely express opinions about Google and Bing, so I’m delighted to have such a hot controversy to incite discussion. Because everyone enjoys a muddle puddle tweetle poodle beetle noodle bottle paddle battle!

Last October, a certain blogger said:

LinkedIn needs to implement some kind of concept extraction to provide a useful topic facet (something I’d also love to see for their regular people search). This is a challenging information extraction problem, especially for the open web, but I also know from experience that it is tractable within a domain. Given LinkedIn’s professional focus, I believe this is a problem they can and should tackle.

Shortly after writing that post, I interviewed at LinkedIn and met Pete Skomoroch, who showed me an early preview of the work his team was doing to make skills a facet for exploring the space of LinkedIn member profiles. That demo made a strong impression on me, giving me a taste of the great products LinkedIn’s data scientists were working on in the lab.

And now I’m delighted that everyone can try out the beta launch of LinkedIn Skills which was announced today at O’Reilly’s Strata 2011 conference on Big Data.

As Pete says in his blog post:

If you search for a particular skill, we’ll surface key people within that community, show you the top locations, related companies, relevant jobs, and groups where you can interact with like minded professionals.  You’ll also be able to explore similar skills and compare their growth relative to each other.

I encourage you to check it out — whether you’re looking for experts on Hadoop, cheese, or anything else! It’s a beta, so I’m sure you’ll find rough edges; but I hope it gives you a sense of how LinkedIn’s data can enable a incredibly powerful and useful exploratory search experience.

No forward-looking statements, except to say that it only gets better from here!

While my blog has always been and will always be a personal one, I do operate under certain constraints as someone whose subject matter relates strongly to his professional interests. I deeply appreciate how long-time readers have respected the balancing act I sometimes have to make as both an independent individual and an employee.

Right now, that means I must respect the conditions of my employer’s quiet period — and I will do so very conservatively (e.g., no Playboy interviews). I apologize if the content of this blog suffers in the interim, but I hope you understand my need to be cautious.

And one last question…are looking for an exciting internship opportunity this summer? Then you’ve come to the right place at the right time: LinkedIn is looking for a few good interns for summer 2011! You can find more details here or go directly to the application form.

If you are interested, I encourage you to act quickly, since we are already interviewing candidates.

“If a man hasn’t discovered something that he will die for, he isn’t fit to live.”

Martin Luther King, Jr. said these words at a speech in Detroit on June 23, 1963. Less than five years later, he died for the cause to which he devoted his life: the advancement of civil rights in the United States and around the world through civil disobedience and other nonviolent resistance.

Today, as Americans commemorate Dr. King’s birthday, there are many ways we can honor his memory and build on his legacy. As much as King advanced the civil rights movement, there is still much to be done to fulfill his dream.

But I’d like to go back to the quote from his speech in Detroit. King’s words reveal a truth even deeper than his struggle for civil rights. They demand that we approach life with passion, that we live to do something more than pass the time.

In the face of pressing day-to-day responsibilities, it is easy to fall into a reactive rhythm, doing what we have to do and then using what time is left to escape into oblivion. For many of us, passion may feel like a nice-to-have, something to think about after we’ve cleared out our queues and gotten a full night of sleep — only to wake up and find that the queue is full again. It is easy to go through life like Sisyphus, sweating profusely as we roll our boulders but lacking the intellectual ambition to question why we make those efforts.

Today, the least we can do to honor the memory of Martin Luther King, Jr. is to reflect on his personal passion to leave the world better than he found it. Hopefully none of us will ever have to make the sacrifice that he made to realize his dream. But if we do not dare to dream at all — if we are not passionate and ambitious about what do — then, indeed, we are not fit to live.

Dare to dream — and live to make that dream a reality.

But I have personal reasons to be interested in Quora. One of their lead engineers, Albert Sheu, was a star intern of mine at Endeca. And Quora raises lots of interesting questions about search, user experience, knowledge management, and online reputation. How could I resist?

I see three potential reasons to use Quora:

1. Objective question answering.
2. Subjective question answering.
3. Community participation.

Let’s consider how Quora fares today on each of these, and where it might go.

1. Objective question answering.

When I blogged about Quora early last year, I said that “I don’t see Quora as a knowledge base of first resort–except possibly to learn more about software startups.” Despite Quora’s recently growth surge, I am not ready to change my answer significantly — I find that Quora’s topics are pretty sparse when I stray from its Silicon Valley focus.

Within that focus, Quora is nailing it. For example, I was curious to learn whether someone who signed a non-compete agreement outside of California was still subject to it if he or she moved to California, where such contracts are legally unenforceable. Not surprisingly, non-compete agreements are a topic on Quora, and I quickly found a useful answer from a lawyer.

But for most objective questions, I’m still turning to Google and Wikipedia — or to Twitter if both of those fail and I am willing to ask a favor of my followers (who kick ass!). Sometimes Google will take me to Quora, but I can’t imagine Quora will succeed through this flow in the long term.

2. Subjective question answering.

I see subjective question answering as Quora’s strongest suit. A good subjective question on Quora — often a “why” question — generates a diverse collection of interesting and informed perspectives. A couple of good example are “Why did Google Wave fail to get significant user adoption?” and “What is lacking in social networking now?“.

Again, these questions are well within the Silicon Valley focus, but I could see Quora extending this value proposition to other verticals if it can grow the communities successfully. And I certainly don’t see myself going to Google or even Twitter to get useful answers to subjective questions. The closest is Topsy, and Quora has the advantage of being explicitly organized around questions and topics.

3. Community participation.

Is Quora a question answering site or a social network? Quora users and employees have tried to answer that question (on Quora, natch), but I’m not sure Quora’s converged enough for anyone to know. What is clear is that Quora emphasizes conversation, making it more like a blog or wiki than an answers site.

Conversation certainly engages its participants. But it also raises the cost of participation. One of the things I love about Google is that it gives me information without unnecessary overhead. When I want conversation, I go to social venues like Twitter.

Perhaps Quora can be both a question answering site and a social network. But I suspect it will need to choose. Most people don’t have the time or patience to participate in additional communities, so question answering is the easier sell to a mass audience. But the participation is what makes Quora especially distinctive today. Perhaps it’s a question of quality vs. quantity.

So, quo vadis, Quora? I suppose I’ll have to check Quora (or Cwora) to find the answers.

I hope everyone I’ve invited is enjoying Quora. But I also hope you take it upon yourselves to circulate more invitations to those who want them. Any Quora user can send out invites — that’s how these viral sites work. If you’re still looking for an invite, I urge you to use Twitter or some other broadcast mechanism to request it. As of today, I will stop responding to Quora invite requests through my blog or email, and I will also delete comments requesting them. I am sorry if this is a bit harsh, but I hope folks understand.