Last night, I had the pleasure to deliver the keynote address as the CIO Summit US. It was an honor to address an assembly of CIOs, CTOs, and technology executives from the nation’s top organizations. My theme was “Science as a Strategy”.

To set the stage, I told the story of TunkRank: how, back in 2009, I proposed a Twitter influence measure based on an explicit model of attention scarcity which proved better than the intuitive but flawed approach of counting followers. The point of the story was not self-promotion, but rather to introduce my core message:

Science is the difference between instinct and strategy.

Given the audience, I didn’t expect this message to be particularly controversial. But we all know that belief is not the same as action, and science is not always popular in the C-Suite. Thus, I offered three suggestions to overcome the HIPPO (Highest Paid Person’s Opinion):

• Ask the right questions.
• Practice good data hygiene.
• Don’t argue when you can experiment!

Asking the Right Questions

Asking the right questions seems obvious — after all, our answers can only be as good as the questions we ask. But science is littered with examples of people asking the wrong questions — from 19th-century phrenologists measuring the sizes of people’s skulls to evaluate intelligence to IT executives measuring lines of code to evaluate programmer productivity. It’s easy for us (today) to recognize these approaches as pseudoscience, but we have to make sure we ask the right questions in our own organizations.

As an example, I turned to the challenge of improving the hiring process. One approach I’ve seen tried at both Google and LinkedIn is to measure the accuracy of interviewers — that is, to see how well the hire / no-hire recommendations of individual interviewers predict the final decisions. But this turns out to be the wrong question — in large part because negative recommendations (especially early ones) weigh much more heavily in the decision than positive ones.

What we found instead was that we should focus on efficiency as an optimization problem. More specifically, there is a trade-off: short-circuiting the process as early as possible (e.g., after the candidate performs poorly on the first phone screen) reduces the average time per candidate, but it also reduces the number of good candidates who make it through the process. To optimize overall throughput (while keeping our high bar), we’ve had to calibrate the upstream filters. How to optimize that upstream filter turns out to be the right question to ask — and one we still continue to iterate on.

More generally, I talked about how, when we hire data scientists at LinkedIn, we look for not only strong analytical skills but also the product and business sense to pick the right questions to ask – questions whose answers create value for users and drive key business decisions. Asking the right questions is the foundation of good science.

Practicing Good Data Hygiene

Data mining is amazing, but we have to watch out for its pejorative meaning of discovering spurious patterns. I used the Super Bowl Indicator as an example of data mining gone wrong — with 80% accuracy, the division (AFC vs. NFC) of the Super Bowl champion predicts the coming year’s stock market performance. Indeed, the NFC won this year (go Giants!) and subsequent market gains have been consistent with this indicator (so far).

We can all laugh at these misguided investors, but we make these mistakes all the time. Despite what researchers have called the “unreasonable effectiveness of data”, we still need the scientific method of first hypothesizing and then experimenting in order to obtain valid and useful conclusions. Without data hygiene, our desires, preconceptions, and other human frailties infect our rational analysis.

A very different example is using click-through data to measure the effectiveness of relevance ranking. This approach isn’t completely wrong, but it suffers from several flaws. And the fundamental flaw relates to data hygiene: how we present information to users infects their perception of relevance. Users assume that top-­ranked results are more relevant than lower-­ranked results. Also, they can only click on the results presented to them. To paraphrase Donald Rumsfeld: they don’t know what they don’t know. If we aren’t careful, a click-­based evaluation of relevance creates positive feedback and only reinforces our initial assumptions – which certainly isn’t the point of evaluation!

Fortunately, there are ways to avoid these biases. We can pay people to rate results presented to them in random order. We can use the explore / exploit technique to hedge against the ranking algorithm’s preconceived bias. And so on.

But the key take-away is that we have to practice good data hygiene, splitting our projects into the two distinct activities of hypothesis generation (i.e., exploratory analysis) and hypothesis testing using withheld data.

Don’t Argue when you can Experiment

I couldn’t resist the opportunity to cite Nobel laureate Daniel Kahneman‘s seminal work on understanding human irrationality. I also threw in Mercier and Sperber’s recent work on reasoning as argumentative. The summary: don’t trust anyone’s theories, not even mine!

Then what can you trust? The results of a well-­‐run experiment. Rather than debating data-­‐free assertions, subject your hypotheses to the ultimate test: controlled experiments. Not every hypothesis can be tested using a controlled experiment, but most can be.

I recounted the story of how Greg Linden persuaded his colleagues at Amazon to implement shopping-cart recommendations through A/B testing, despite objections from a marketing SVP. Indeed, his work — and Amazon’s generally — has strongly advanced the practice of A/B testing in online settings.

Of course, A/B testing is fundamental to all of our work at LinkedIn. Every feature we release, whether it’s the new People You May Know interface or improvements to Group Search relevance, starts with an A/B test. And sometimes A/B testing causes us to not launch — we listen to the data.

Don’t argue when you can experiment. Decisions about how to improve products and processes should not be by an Oxford-­style debate. Rather, those decisions should be informed by data.

Conclusion: Even Steve Jobs Made Mistakes

Some of you may think that this is all good advice, but that science is no match for an inspired leader. Indeed, some pundits have seen Apple’s success relative to Google as an indictment of data-­driven decision making in favor of an approach that follows a leader’s gut instinct. Are they right? Should we throw out all of our data and follow our CEOs’ instincts?

Let’s go back a decade. In 2002, Apple faced a pivotal decision – perhaps the most important decision in its history. The iPod was clearly a breakthrough product, but it was only compatible with the Mac. Remember that, back in 2002, Apple had only a 3.5% market share in the PC business. Apple’s top executives did their analysis and predicted that they could drive the massive success of the iPod by making it compatible with Windows, the dominant operating system with over 95% market share.

Steve Jobs resisted. At one point he said that Windows users would get to use the iPod “over [his] dead body”. After continued convincing, Jobs gave up. According to authorized biographer Walter Isaacson, Steve’s exact words were: “Screw it. I’m sick of listening to you assholes. Go do whatever the hell you want.” Luckily for Steve, Apple, and the consumer public, they did, and the rest is history.

It isn’t easy being one those ass­holes. But that’s our job, much as it was theirs. It’s up to us to turn data into gold, to apply science and technology to create value for our organizations. Because without data, we are gambling on our leaders’ gut feelings. And our leaders, however inspired, have fallible instincts.

Science is the difference between instinct and strategy.

Recently I had a couple of great opportunities to share my thoughts publicly, and I wanted to make sure readers here were aware of them.

The first was a special guest appearance on The Semantic Link, a program hosted by Paul Miller with regular panelists Peter Brown, Christine Connors, Eric Franzon, Eric Hoffer, Bernadette Hyland, and Andraz Tori. It was a lot of fun, and a great warm-up for the keynote I’ll be delivering on “Scale, Structure, and Semantics” at the upcoming Semantic Tech & Business Conference (SemTechBiz), which will take place in San Francisco in June.

The second was a live interview on Internet Evolution, hosted by Mary Jander and Nicole Ferraro. They clearly did their homework, scouring my blog posts and web commentary for everything controversial I’d ever said — and then some! If that’s enough to pique your interest, then I encourage you to listen to the recorded interview and read the chat transcript at Internet Evolution.

Happy to answer questions based on either of these sessions — comment away!

Note: This post was written by Yael Garten, a Senior Data Scientist at LinkedIn. Yael joined Linkedin in 2011, where she leads our mobile analytics team. She previously worked at Stanford on text mining, personalized medicine, and biomedical informatics.

We live in an era of Big Data. But how do we use all of that data to answer questions and communicate those answers effectively?

My colleagues and I at LinkedIn were fortunate enough to hear answers from Noah Iliinsky, who literally wrote the book on designing data visualization.

Earlier this month, we hosted Noah at LinkedIn to give a tech talk on “Designing Effective Data Visualizations“. We are proud to make these tech talks open to the public, and enjoyed a great mix of attendees from local companies and universities. If you couldn’t attend the talk in person or remotely, I encourage you to watch the recording, embedded above.

Why do we visualize data? As Noah tells us, visualization makes data accessible. It gives us faster access to actionable insights and allows access to huge amounts of data. Visualization enables both data exploration (when you are still trying to discover the story) and data explanation (when you have a story to tell). Noah reviewed some great examples (watch the talk!), with an emphasis on the dos and don’ts of data visualization.

In particular, he provided a step-by-step framework for traversing the path from question to answer:

Phase 1: Decide what to visualize.

• Understand the question your audience wants to answer.
• Understand the actions they are hoping the answer will drive.
• Consider who is consuming this data — their needs, biases, etc.
• Decide what data to use — and what data not to use — and what relationships you are interested in.
• Explore the data and construct a storyline.

Phase 2: Decide how to visualize it.

• Use appropriate visual encodings for data and relationships (cf. http://complexdiagrams.com/properties).
• Limit the data you include.
• Use position for your most important relationship.
• Try different axes.
• Show your visualization to different people, without explanations. Show an expert, show a layman.
• Iterate, iterate, iterate!

Noah also shared his thoughts on how to visualize social networks. He recommended useful tools for data visualization, including Tableau, Spotfire, D3, Processing, ggplot2, Omnigraffle, and OmnigraphSketcher.

• You are not your audience. This is a huge lesson that all of us must internalize to be great at what we do. Consider what you need to communicate to marketers, investors, member of the general public, etc.
• Do user research! Understand your users’ hopes, dreams, and favorite flavors! Understand their identity, their jargon, culture, etc.
• Remember that your success is defined by your customer’s success. If you can’t satisfy your customer’s needs, you have failed — no matter how insightful your analysis.

You can enjoy the talk by watching the embedded video above. And you can find more LinkedIn tech talks on our YouTube channel.

One of the highlights of the recent Data 2.0 Summit was a panel featuring:

• Alexander Gray, CTO of SkyTree
• Anthony Goldbloom, CEO of Kaggle
• Josh Wills, Director of Data Science at Cloudera

The focus of the panel was supposed to be about “Data Science and Predicting the Future”, but the most contentious topic was whether data, algorithms or people (that is, the data scientists themselves) were the most important factor in the practice and success of data science.

Yes, we one-upped the debate that my colleague Monica Rogati instigated at this year’s Strata conference. In fact, Josh cited the “better data beats more data beats clever algorithms” argument that Monica made in her own Strata presentation. And, just like at Strata, there was a healthy dose of audience participation.

Of course, I came down on the side of data — which I believe won the debate hands down.

I’m a fan of clever algorithms, which Alexander had to defend given that Skytree’s core value proposition is better machine learning algorithms delivered at scale. But I’m with Peter Norvig et al. on the dominance of data over algorithms.

Favoring data over people was a harder choice. Anthony naturally made the case for people (Kaggle’s claim to fame is assembling many of the world’s best data scientists by organizing competitions). Hopefully my team won’t quit en masse when they read this blog post! But I think they’ll agree with me that, without the incredible data we work with at LinkedIn, they’d be unable to deliver the awesomeness that I’ve come to expect from them.

There’s a saying that we all cook from the same cookbooks, so that it’s the ingredients that make all the difference. To take the metaphor further, you can also try to poach your rival’s chefs. But data is the biggest entry barrier — and the most sustainable competitive advantage.

Of course, we should have the best people apply the best algorithms to work with the best data. But data comes first. The best meal starts with the best ingredients.

The video of the presentation that Claire Hunsaker and I delivered on “Humans, Machines, and the Dimensions of Microwork” at Strata 2012 is now available as part of the complete video compilation. I’ve taken the liberty to upload it to YouTube — feel free to watch the embedded video above.

I’ll be participating in the Data 2.0 Summit on Tuesday, April 3rd, and I hope to see some of you there. Last year, my colleague (and fellow LinkedIn data scientist) Scott Nicholson attended and wrote this guest post about it. This year, I’m not only attending but participating on a panel about social data, moderated by AllthingsD Senior Editor Liz Gannes.

There’s a great line-up of speakers for the day, including:

• Bram Cohen, the founder, chief scientist, and inventor of BitTorrent, the leading peer-to-peer file sharing protocol for sharing large files on the Internet.
• Michael Driscoll, CTO and co-founder of the Metamarkets Group. He moderated a fantastic debate at the recent Strata conference about the relative importance of domain expertise or machine learning for data scientists.
• Gil Elbaz, the founder and CEO of Factual, an information marketplace. He is also the co-founder of Applied Semantics, which Google acquired in 2003 for $102M and turned into the foundation for AdSense (now a $10B business).
• Anthony Goldbloom, co-founder and CEO of Kaggle,  a platform for data science competitions that generated a lot of discussion at Strata.
• Stefan Weitz, director of search at Bing. He’ll be on my panel. Also see the discussion I had with him in the comment thread for a post on “Why Are People So Clueless About Search?“.

And lots more, but you get the idea. I’m thrilled to be part of such a talent-heavy program and looking forward to insightful discussions with with fellow panelists and attendees. Also a great excuse to spend a day in the city (note for my former townspeople — that’s what they call San Francisco around here).

Conventional wisdom holds that physical compliments are counter-productive as pick-up lines. Indeed, a dating site did some analysis showing a negative correlation between such compliments and the probability of a positive response.

But, as m6d Chief Scientist and 3-time KDD Cup winner Claudia Perlich explained in her recent talk at LinkedIn, we have to watch out for confounding variables. In the dating scenario above, beauty is a confounding variable: it determines both the probability of getting a positive response and of the probability of a suitor offering physical compliments. Hence, we need to control for the actual beauty or it can appear that making compliments is a bad idea.

Perlich does not work on online dating, but rather in the data-driven world of online advertising. Specifically, she and her team work on real-time bidding optimization.

Perlich described a variety of design choices that have general applicability to data science problems. For example, her team used hashed tokens of previously visited URLs, rather than the URLs themselves, as features for their machine learning models. They avoided the use of personally identifying information (PII) or even demographic information about their users. These decisions were counterintuitive — typically, more data leads to better results. But Perlich found that these restrictions did not sacrifice accuracy, and had the further benefit of keeping their approach general rather than application- or customer-specific.

Perlich also described several technical challenges that her team had to overcome. For example, they found they could not sample users, so they instead sampled events — that is, visits, impressions, and conversions. They also found that their linear models tended to suffer from overfitting in their top predictions — a problem they resolved by introducing a spline model.

The talk was deeply technical and yet very relevant and accessible to a broad audience of data scientists and engineers. There’s much more content than fits in this small summary, so I encourage you to watch the video! And you can watch more LinkedIn tech talks here.

In the past few years, prosopagnosia, also known as “face blindness”, has received a fair amount of attention from researchers, as well as from the popular press.

My first exposure to the topic was Joshua Davis’s article entitled “Face Blind“, which appeared in Wired in November 2006. I was intrigued, especially since I’d long recognized that I had difficulty recognizing people by face. Perhaps the person who has done most to raise awareness of prosopagnosia is neurologist Oliver Sacks, who has prosopagnosia himself.

The Wired article inspired me to explore the subject. I discovered faceblind.org and found quizzes that tested for prosopagnosia. On one of these, where random guessing would have earned a score of 50%, I scored in the low 60s. My initial reaction was that my score wasn’t so bad — it was a hard test! Then my wife took the test and scored in the high 90s. That’s when I realized that I didn’t just have difficulty recognizing faces — I was almost incapable of it.

Faced with this realization, I had to decide whether to share it with my friends and family, let alone with my broader set of social and professional acquaintances. It was tempting not to — after all, why tell the world that I wasn’t “normal”?

But eventually I realized that it would be better for people around me to know than not know. The biggest downside to prosopagnosia isn’t the momentary embarrassment of not recognizing someone — it’s the content fear of offending people who may think you don’t value them enough to recognize or acknowledge them.

Hence, I spread the word through my colleagues, ensuring that most of the people with whom I interacted regularly would find out without any big announcements. Some of my co-workers were surprised, since I do a pretty good job of recognizing people using non-facial clues — height, hair, clothing, where I run into them, etc. I have a great memory, and I have no problems with voice recognition. In other words, I have lots of work-arounds.

Fortunately, I work with a lot of people who understand machine learning — which is a great framework for understanding how I recognize people. I simply work with a different set of features than most people, but fortunately I achieve sufficient precision and recall to pass as “normal” most of the time.

Anyway, if you didn’t already know that I had prosopagnosia, welcome to the inner circle! And if you ever felt that I walked by you without recognizing or acknowledging you, please accept my belated apology.

Finally, if you’re curious to learn more about prosopagnosia, I encourage you to watch the 60 Minutes segments above.

Just watch the presentation, which stole the show at Strata 2012. The written word cannot do justice to Avinash’s passion and his extraordinary ability to communicate it.

I spent the last three days at the O’Reilly Strata Conference, an assembly of two thousand over 2500 people focused on data science and its applications. While I’m wary of industry conferences from attending vendor-fests in my past life in the enterprise software world, Strata is an exceptionally good conference. The speakers were a who’s who of data science, including Lucene and Hadoop creator Doug Cutting, search user interface pioneer Marti Hearst, and Google chief economist Hal Varian. You can find the tweet stream for the conference at hash tag #strataconf.

Tuesday

I spent Tuesday in the Deep Data session, billed as a no-holds-barred program for data scientists. My two favorite talks:

• Claudia Perlich, winner of three KDD cups, talked about using information to pick the right action and to influence people such that they behave in a way that is better for them, better for us, and possibly better for society in general.
• Monica Rogati, my colleague at LinkedIn and the epitome of a data scientist, delivered a fantastic talk about machine learning models and training data in the real world, extending Peter Norvig‘s point about the “unreasonable effectiveness of data” to observe that more data beats clever algorithms but better data beats more data.

But the most fun that day was the Oxford-style debate featuring Drew Conway, Pete Skomoroch, Mike Driscoll, DJ Patil, Amy Heineike, Pete Warden, and Toby Segaran. The question proposed was absurdly Manichean: if you had to hire your first data scientist and could only hire one, would you pick a domain expert or a machine learning expert? After the moderator suppressed some initial attempts to hedge (“both”, “it depends”, etc.), the debaters ripped into the question by taking extreme positions and defending them with gusto. It was a lot of fun, with enthusiastic audience participation and the debaters exploiting their inside knowledge of their opponents’ work histories. In the end, the machine learning side won by a small margin.

I then had the good fortune to grab dinner with Marti Hearst and Hal Varian at Xanh – a wonderful mix of great food and conversation.

Wednesday

The Wednesday morning keynote session offered some gems:

• Cloudera CEO Mike Olson urged big data practitioners to focus on guns, drugs, and oil.
• Doctor and data geek Ben Goldacre delivered a mesmerizing and disturbing talk about the suppression of inconvenient medical trial results and analytical tools to discover it.

But the person who stole the show was Google’s Avinash Kaushik, who talked about making love with data to find orgasm-inducing actions to change the world and make more money. Unfortunately this was the one talk that was not recorded, but you can read the summary on Avinash’s Google+ page.

As a speaker, I held “office hours” on Wednesday. It was supposed to be a 40-minute slot for conference attendees to come and ask me question. But somehow those 40 minutes extended into three hours of conversation about everything from normalized KL divergence to interview problems — and segued into a reception with specialty big-data cocktails. By the time I got back to my apartment, my voice, brain, and liver were spent.

Thursday

I spent most of Thursday morning in the speaker lounge, recovering from the previous evening and making last touches on my presentation. But I couldn’t resist attending a two-part session on privacy. Indeed, this session was distinctive enough to merits it’s own hash tag: #strataprivacy.

The first part featured O’Reilly’s Alex Howard moderating Intelius Chief Privacy Officer Jim Adler and NYU PhD student Solon Barocas on a panel provocatively titled  ”If Data Wants to Be Free, is Privacy a Prison?” It was a great discussion, and I enjoyed the opportunity to offer my own provocative question through Twitter. Since the panelists were arguing that it was unethical to infer private facts from public data, I asked if they were trying to establish a new form of thoughtcrime.

The second panel, entitled “Pretty Simple Data Privacy“, featured Kaitlin Thaney from Digital Science, Betsy Masiello from Google, and John Wilbanks from the Kauffman Foundation for Entrepreneurship. Given that today was the first day of Google’s new privacy policy, there was no avoiding focus on the associated controversy. I did try to get Betsy to address my charge that Google doesn’t think users own their search history (cf. “Google vs. Bing: A Tweetle Beetle Battle Muddle“), but she said she was unfamiliar with the details of that event. I do wish that someone at Google with more familiarity would respond publicly.

Back to the speaker room after lunch, until my own talk with Samasource’s Claire Hunsaker on “Humans, Machines, and the Dimensions of Microwork“. I’ll post the slides (and there will be a video on the conference site), but the sound bite is that you need to keep crowdsourcing tasks simple, manage the trade-off between task value and difficulty, and watch out for systematic bias.

I wrapped up the conference by hearing William Gunn talk about how Mendeley is disrupting bibliometrics and perhaps the entire academic publishing and reputation ecosystem. I laud his ambition and wish him and Mendeley luck in this quest.

In summary, three days of great talks, conversations, and general enjoyment. My thanks to Strata organizers Edd Dumbill and Alistair Croll for putting together such an outstanding event and for giving me the opportunity to participate.

My excursion to Seattle was delightful, and I thought I’d share some details with readers.

I spent most of Friday at the University of Washington, meeting with graduating PhD students.  I’ve always known that UW is a top school, but I was particularly impressed with this batch. I was pleasantly surprised to see folks like Nodira Khoussainova and Kayur Patel working to bring together the often disparate worlds of databases, machine learning, and HCI in order to make people more effective at solving “big data” problems. I realize that I’m aiding and abetting other employers with whom I compete for top talent, but it would be wrong not to encourage everyone to find worthy challenges for these budding scientists.

I then went to the Space Needle to meet up with the WSDM 2012 crowd. Jaime Teevan and Eytan Adar outdid themselves, providing a great setting for folks to mingle, imbibe, and enjoy a spectacular view of Seattle.

Saturday I attended the “social” day of the WSDM conference.

Andrew Tomkins chaired the first morning session, which included Hila Becker‘s latest work on identifying event content in social media and Georgios Zervas presenting the work on the analyzing reputational effects of Groupon that triggered quite a controversy last September. After the break came the spotlight section — a great sequence of 5-minute presentations that in which researchers both summarized their contributions and lured attendees to visit their posters. I hope that more conferences adopt this format, which optimizes for communicating ideas and discourages long-winded expositions.

I then had the pleasure to have lunch with Jan Pedersen and friends at Blueacre Seafood — great food and even better conversation. We both noted the irony that, even though we are practically neighbors, we only seems to meet up at events like these..

I made it back to the conference in time to hear the two best-paper awardees: Adam Sadilek on “Finding Your Friends and Following Them to Where You Are” and Yaron Singer on “How to Win Friends and Influence People, Truthfully: Influence Maximization Mechanisms for Social Networks“. I highly recommend both papers, especially if you are interested in either social network prediction or the underlying economics of influence.

Another coffee break, and then the keynote: Hilary Mason on “The Secret Life of Social Links”. Hilary is a great speaker — I first met her when I invited her to the Workshop on Search and Social Media (SSM 2010) at WSDM 2010. She didn’t disappoint, and it’s great to see practitioners like her crossing the aisle to engage the academic community. Not to mention infusing their slides with lolcats.

The conference wrapped up at 5pm, but then we bussed over to Microsoft Research for the Social Search Social. That was a fun event designed to cross-pollinate the WSDM and CSCW communities. Meredith Ringel Morris, Gene Golovchinksy, Jeremy Pickens, Madhu Reddy, Chirag Shah, and Michael Twidale put together a great program of 45-second madness presentations and “speed-dating” to pair up WSDM and CSCW attendees. It was far too short, but a lot of fun. And some of us kept up the social spirit by grabbing dinner afterward at Blue C Sushi.

To everyone I met in the last couple of days: thanks for the great company and conversation! Keep sharing ideas and making data and science social.

First, I wanted to give readers a heads up that I’ll be in Seattle this Friday and Saturday. I’ll spend Friday afternoon at the University of Washington, meeting with some of their outstanding computer science doctoral students. My schedule filled up with unexpected haste! But if you’re on campus and urgently want to meet, let me know and I’ll see what I can do.

Saturday I’ll be attending the social track of WSDM 2012, the premier international ACM conference covering research in the areas of search and data mining on the Web. I’m excited about the program, as well as the opportunity to catch up with friends and make new ones. Back in 2010, I had the pleasure of co-organizing the Workshop on Search and Social Media (SSM 2010) and being the official ACM blogger for WSDM 2010. You can read my posts here.

Then, on Saturday evening, I’ll be heading to Microsoft Research to attend the Social Search Social (SSS 2012). Hats off to organizers Meredith Ringel Morris, Gene Golovchinksy, Jeremy Pickens, Madhu Reddy, Chirag Shah, and Michael Twidale for creating what looks to be a fun (and very social!) event. I’m especially looking forward to the 45-second “madness” presentations (in which I’m participating) and the “speed dating” to help cross-pollinate  the WSDM and CSCW communities.

Hope to see some of you there, and of course will share what I learn here at The Noisy Channel. I also encourage you to follow the tweet streams for #wsdm2012 and #sss2012.

As regular readers know, I have a deep affection for Carnegie Mellon University, where I did my graduate work. I’m happy to announce that two of my colleagues (both fellow CMU PhDs) will be giving talks at CMU in a couple of weeks, and I hope that some of you will have the opportunities to attend.

On Tuesday, February 7th, Abhimanyu Lad will be hosting an information session at 6pm in Scaife Hall, Room 214. Abhi is rock star on our data science team, and he’s been working on the next generation of LinkedIn search. You can get a taste of his work from his recent HCIR 2011 presentation, “Is it Time to Abandon Abandonment?“. Abhi will talk about a variety of technical challenges that data scientists and engineers are working on at LinkedIn.

On Thursday, February 9th, Paul Ogilvie will talk about “Where Big Data Meets Real-Time: Efficiently Indexing and Ranking News using Activity” at 3:30pm in GHC 6115. Paul is responsible for article relevance infrastructure and algorithms on LinkedIn Today, a great example of social navigation – not to mention a great success for users. Paul will talk about the technical details that make LinkedIn Today possible, including a novel use of inverted lists to efficiently index and support real-time updates to document representations.

And, even if you can’t make it to the talks, I encourage you to visit the LinkedIn booth at the EOC fair on Wednesday, February 8th. We’re looking for great software engineers and data scientists, and we’re especially interested in interns.

This is the season of annual reviews, at least at LinkedIn. Performance reviews can be daunting for both employees and managers — at least everywhere that I’ve worked. Not only are we as human beings terrible at delivering feedback, but we also receive bad advice as managers.

For example, many of us have learned the “feedback sandwich” method, a technique that doesn’t hold up to scientific validation. Watch the video below to see what Stanford professor Clifford Nass has learned from his experiments (see my review of his book here).

Here is what I suggest as a format for performance feedback, whether for writing your own self-assessment or delivering feedback to reports or peers on their performance:

1) What is your day job?

Everyone needs a day job — a mission with a crisp set of responsibilities and deliverables. If you don’t know what you’re responsible for delivering, you can’t assess how well you are delivering it. You should know and articulate your top priorities — at most three, with a clear #1. For further reading, I suggest the Quora discussion on OKRs (Objectives and Key Results), an idea pioneered by Intel and now used at top technology companies (including LinkedIn and Google).

2) How are you performing in your day job?

Hopefully you make more contributions than you can count. But make sure that your day job comes first. If you find that a disproportionate fraction of your contribution is outside your day job, then consider changing your day job. Your top priority is to meet (hopefully exceed!) the expectations for your day job — expectations you should set early and revisit regularly. Performance reviews are a great opportunity to brag.

3) What do you do beyond your day job?

Your day job should be strongly aligned with your team and company’s top priorities. But great employees contribute beyond their day job towards other team and company priorities. For example, talent is our top priority at LinkedIn, so we particularly value contributions to hiring and growing our talent. And, at least in every environment I’ve experienced, the best employees are those who help make others successful.

4) How do you want to grow?

This is really a two-part question. First, what do you want to do next? That could mean getting better at your day job, evolving your current responsibilities, or taking on a different role. Second, what are you doing to get there? You are ultimately responsible for your own professional development. But one of your manager’s top responsibilities is to help you identify and advance along the path that is best for you. And performance reviews are a great opportunity to make you think about the future.

Regardless of how your company manages performance, these are the key questions you should think about. Performance feedback is a great opportunity to focus on professional development — your own and that of the people you work with everyday. Make the most of it!

Let me preface this post by saying that this is my personal blog, and that my opinions here are not necessarily those of my employer.

With that out of the way, I love the premise of Hitch.me: a dating site for professionals based on LinkedIn. I won’t confirm or deny the number of my colleagues who have thought about building a dating site based on our data, but it’s great to see someone using our APIs to do so. And the marketing video, while not exactly politically correct, is brilliant.

Yet another reason to work as a data scientist at LinkedIn!

Information Scent

In the early 1990s, PARC researchers Peter Pirolli and Stuart Card developed the theory of information scent (more generally, information foraging) to evaluate user interfaces in terms of how well users can predict which paths will lead them to useful information. Like many HCIR researchers and practitioners, I’ve found this model to be a useful way to think about interactive information seeking systems.

Specifically, faceted search is an exemplary application of the theory of information scent. Faceted search allows users to express an information need as a keyword search, providing them with a series of opportunities to improve the precision of the initial result set by restricting it to results associated with particular facet values.

For example, if I’m looking for folks to hire for my team, I can start my search on LinkedIn with the keywords [information retrieval], restrict my results to Location: San Francisco Bay Area, and then further restrict to School: CMU.

Precision / Recall Asymmetry

Faceted search is a great tool for information seeking systems. But it offers a flow that is asymmetric with respect to precision and recall.

Let’s invert the flow of faceted search. Rather than starting from a large, imprecise result set and progressively narrowing it; let’s start from a small, precise result set and progressively expand it. Since faceted search is often called “guided navigation” (a term Fritz Knabe and I coined at Endeca), let’s call this approach “guided exploration” (which has a nicer ring than “guided expansion”).

Guided exploration exchanges the roles of precision and recall. Faceted search starts with high recall and helps users increase precision while preserving as much recall as possible. In contrast, guided exploration starts with high precision and helps users increase recall while preserving as much precision as possible.

That sounds great in theory, but how can we implement guided exploration in practice?

Let’s remind ourselves why faceted search works so well. Faceted search offers the user information scent: the facet values help the user identify regions of higher precision relative to his or her information need. By selecting a sequence of facet values, the user arrives at a non-empty set that consists entirely or mostly of relevant results.

How to Expand a Result Set

How do we invert this flow? Just as enlarging an image is more complicated than reducing one, increasing recall is more complicated than increasing precision.

If our initial set is the result of selecting multiple facet values, then we may be able to increase recall by de-selecting facet values (e.g., de-selecting San Francisco Bay Area and CMU in my previous example). If we are using hierarchical facets, then rather than de-selecting a facet value, we may be able to replace it with a parent value (e.g., replacing San Francisco Bay Area with California). We can also remove one or more search keywords to broaden the results (e.g., information or retrieval).

Those are straightforward query relaxations. But there are more interesting ways to expand our results:

• We can replace a facet value with the union or that value and similar values (e.g., replacing CMU with CMU OR MIT).
• We can replace the entire query (or any subquery) with a union of that query and the results for selecting a single facet value (e.g., ([information retrieval] AND Location: San Francisco Bay Area AND School: CMU) OR Company: Google)
• We can replace the entire query (or any subquery) with a union of that query and the results for a keyword search a single facet value (e.g., ([information retrieval] AND Location: San Francisco Bay Area AND School: CMU) OR [faceted search]).

As we can see, there are many ways to progressively refine a query in a way that expands the result set. The question is how we provide users with options that  increase recall while preserving as much precision as possible.

Frequency : Recall :: Similarity : Precision

Developers of faceted search systems don’t necessarily invest much thought into deciding which faceted refinement options to present to users. Some systems simply avoid dead ends, offer user all refinement options that least to a non-zero result set. This approach breaks down when there are too many options, in which case most systems offer users the most frequent facet values. A chapter in my faceted search book discusses some other options.

Unfortunately, the number of options for guided exploration – at least if we go beyond the very limited basic options — is too vast to apply such a naive approach. Unions never lead to dead ends, and we don’t have a simple measure like frequency to rank our options.

Or perhaps we do. A good reason to favor frequent values as faceted refinement options is that they tend to preserve recall. What we need is a measure that tends to preserving precision when we expand a result set.

That measure is set similarity. More specifically, it is the asymmetric similarity between a set and a superset containing it, which we can think of as the former’s representativeness of the latter. If we are working with facets, we can measure this similarity in terms of differences between distributions of the facet values. If the current set has high precision, we should favor supersets that are similar to it in order to preserve precision.

I’ll spare readers the math, but I encourage you to read about Kullback-Leibler divergence and Jensen-Shannon divergence if you are not familiar with measures of similarity between probability distributions. I’m also glossing over key implementation details  – such as how to model distributions of facet values as probability distributions, and how to handle  smoothing and normalization for set size. I’ll try to cover these in future posts. But for now, let’s assume that we can measure the similarity between a set and a superset.

Guided Exploration: A General Framework

We now have the elements to put together a general framework for guided exploration:

• Generate a set of candidate expansion options from the current search query using operations such as the following:
• De-select a facet value.
• Replace a facet value with its parent.
• Replace a facet value with the union of it and other values from that facet.
• Remove a search keyword.
• Replace a search keyword with the union of it and related keywords.
• Replace the entire query with the union of it and a related facet value selection.
• Replace the entire query with the union of it and a related keyword search.
• Evaluate each expansion option based on the similarity of the resulting set to the current one.
• Present the most similar sets to the user as expansion options.

Visualizing Drift

It’s one thing to tell a user that two sets are distributionally similar based on an information-theoretic measure, and another to communicate that similarity in a language the user can understand. Here is an example of visualizing the similarity between [information retrieval] AND School: CMU and [information retrieval] AND School: (CMU or MIT):

As we can see from even this basic visualization, replacing CMU with (CMU OR MIT) increases the number of results by 70% while keeping a similar distribution of current companies — the notable exception being people who work for their almae matres.

Conclusion

Faceted search offers some of the most convincing evidence in favor of Gary Marchionini‘s advocacy that we “empower people to explore large-scale information bases but demand that people also take responsibility for this control”. Guided exploration aims to generalize the value proposition of faceted search by inverting the roles of precision and recall. Given the importance of recall, I hope to see progress in this direction. If this is a topic that interests you, give me a shout. Especially if you’re a student looking for an internship this summer!

Every year brings its own adventures, but for me 2011 will be a tough act to follow.

A year ago, I’d just started working at LinkedIn, and my biggest concern was selling our apartment in Brooklyn so that my family could join me in California.

Little did I imagine that my new manager, who had just recruited me from Google to LinkedIn (and persuaded my family to change coasts!), would leave three months later for a startup. Welcome to Silicon Valley! At the time, I felt unready for the abrupt transition into the product executive team. In retrospect, I’m thankful for the kick in the pants that helped me transform my role and brought the best out of a great team.

Summer brought the excitement of LinkedIn’s IPO. The process was exhilarating, especially to someone who had worked for over a decade at a pre-IPO company.

Nonetheless, we didn’t let the IPO distract us from our mission. In March, we celebrated our 100 millionth member; by November, we passed 135 million. And lots more. We released new data products like Skills and Alumni. We won the OSCON Data Innovation Award for contributions to the open source software for big data. We also acquired a few companies, including search engine startup IndexTank. In short, we heeded the two short words on the back of our commemorative IPO t-shirts: “next play”.

Fall was an intense season of conferences. Between CIKM, HCIR, RecSys, Strata, and a talk at CMU, it was a great opportunity to connect and reconnect with researchers and practitioners around the world. I am particularly proud of the success of this year’s HCIR workshop, which showed how much the workshop (now to become a 2-day symposium!) has grown up in five years.

But what capped my year off was seeing Endeca, the company I helped start in 1999, become one of Oracle’s largest acquisitions. Even though it’s been two years since I left, Endeca will always be a core facet of my professional identity. I look forward to great things from all the folks I worked with.

That brings us to 2012, ready to start a new year of adventures. Tough or not, our job is to make every new year more amazing than the previous ones. I’m ready for the challenge, and I hope you are too.

Here’s a teaser of what I have planned:

• My team at LinkedIn is launching into 2012 with a strong focus on derived data quality and relevance. As regular readers know, I see data quality and richer interfaces for information seeking as inseparable concerns. And, speaking of quality, we’re hiring!
• I’ll be speaking at Strata in a couple of months with Claire Hunsaker of Samasource about “Humans, Machines, and the Dimensions of Microwork“. I’m very excited to talk about the intersection of crowdsourcing and data science. And I’ll be joined by three of LinkedIn’s top data scientists: Monica Rogati, Sam Shah, and Pete Skomoroch.
• I’ll be co-chairing the RecSys Industry Track this fall with Yehuda Koren. I’m honored to have the opportunity to work with Yehuda, who was part of the Netflix Grand Prize team and won best paper at RecSys 2011. We’re still putting together the program, but you can look at last year’s program to get an idea of what’s in store.
• I’ll be at the CIKM Industry Event, this time as an invited speaker. CIKM will be take place in Maui this fall and I’m excited about the program that Evgeniy Gabrilovich is putting together for the Industry Event. It will be an all-invited program, just like last year.

I hope you’re also starting 2012 with a fresh sense of purpose. Let’s take a last moment to reflect on a great 2011, and then…NEXT PLAY!

Better yet, you can now, for the first time in the workshop’s history, watch videos of the presentations. Embedded below are videos of Gary Marchionini’s keynote address and of the two paper presentation sessions. Thanks again to Google for being such a gracious host — now online as well as offline!

Keynote

Morning Presentations

Afternoon Presentations

Privacy is the third rail of the cloud. On one hand, the ease of sharing information and the power of analytics have produced extraordinary value for consumers, as well as great business models for companies that serve those consumers. On the other hand, people have good reason to worry about the unintended consequences of over-sharing.

When I attended the O’Reilly Strata New York Conference in September, I had the pleasure to hear and meet Intelius’s Jim Adler talk about being his company’s “accidental chief privacy officer”. Intelius‘s main product is people search — an area that naturally brings up privacy concerns. Especially since Intelius aggregates and publishes information about people from databases of public records, eroding a history of “privacy through difficulty“. Impressed with Jim’s talk at Strata, I persuaded him to deliver a similar talk at LinkedIn, the video of which you can find above. You can also find his slides on SlideShare.

Jim brings nuance to the discussion of privacy — nuance that discussions of online privacy often lack. For example, he responded to the recent controversy about social networks’ “real names” policy with a measured post entitled “Nyms, Pseudonyms, or Anonyms? All of the Above“.

Jim appropriately opened his talk by disclosing a personal example. He shares his name with a more prominent personal injury lawyer who dominates search results for that name, raising the potential of taint by association. Intelius’s core technical problem is to cluster inputs from the sources it aggregates, thus mapping each person to exactly one record in its database.

Jim went on to note that we are at a stage in the privacy debate where we are likely to see more regulation. He makes a few key observations:

• Social norms, which form the basis of our laws and regulations (the notion of a “reasonable expectation of privacy) have changed suddenly, leading to a “privacy vertigo” where suddenly the whole world now feels like a small town.
• Sharing is a gateway from private to public, which often leads to violation of expectations. This problem is not new, but the efficiency of online sharing dramatically amplifies the unintended consequences of sharing. It is crucial that the parties involved in sharing data also have shared expectations around how that data will be used or disclosed.
• We need to distinguish between data use and data access, and not to try to regulate data use with data access regulations. He cites the Fair Credit Reporting Act as one of the most inspired laws of the last 40 years to regulate data use. If you don’t have time to listen to the whole talk, I recommend you jump to 25:12, where he discusses this law in detail.

There’s a lot more in the talk, so I’m not going to try to summarize it all here. I strongly encourage you to check out the video (which includes lengthy Q&A) and the slides. Better yet, let’s use the comments to discuss!

I’ve posted slides and summaries for all ten CIKM 2011 Industry Event presentations:

• Stephen Robertson (Microsoft Research): Why Recall Matters
• John Giannandrea (Google): Freebase – A Rosetta Stone for Entities
• Jeff Hammerbacher (Cloudera): Experiences Evolving a New Analytical Platform: What Works and What’s Missing
• Khalid Al-Kofahi (Thomson Reuters): Combining Advanced Technology and Human Expertise in Legal Research
• Chavdar Botev (LinkedIn): Databus: A System for Timeline-Consistent Low-Latency Change Capture
• Ben Greene (SAP): Large Memory Computers for In-Memory Enterprise Applications
• David Hawking (Funnelback): Search Problems and Solutions in Higher Education
• Ed Chi (Google): Model-Driven Research in Social Computing
• Vanja Josifovski (Yahoo! Research): Toward Deep Understanding of User Behavior on the Web
• Ilya Segalovich (Yandex): Improving Search Quality at Yandex: Current Challenges and Solutions

This post is last in a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

The final talk of the CIKM 2011 Industry Event was a talk from Yandex co-founder and CTO Ilya Segalovich on “Improving Search Quality at Yandex: Current Challenges and Solutions“.

Yandex is the world’s #5 search engine. It dominates the Russian search market, where it has over 64% market share. Ilya focused on three challenges facing Yandex: result diversification, recency-specific ranking, and cross-lingual search.

For result diversification, Ilya focused on queries containing entities without any addition indicators of intent. He asserted that entities offer a strong but incomplete signal of query intent, and in particular that entities often call for suggested query reformulations. The first step in processing such a query is entity categorization. Ilya said that Yandex achieved almost 90% precision using machine learning, and over 95% precision by incorporating manually tuned heuristics. The second step is enumerating possible search intents for the identified category in order to optimize for intent-aware expected reciprocal rank. By diversifying entity queries, Yandex reduced abandonment on popular queries, increased click-through rates, and was able to highlight possible intents in result snippets.

Ilya then talked about the problem of balancing recency and relevance in handling queries about current events. He sees recency ranking as a diversification problem, since a desire for recent content is a kind of query intent. A challenge is managing recency-specific ranking is to predict the recency sensitivity of the user for a given query. Yandex considers factors such as the fraction of results found that are at most 3 days old, the number of news results, spikes in the query stream, lexical cues (e.g., searches for “explosion” or “fire”), and Twitter trending topics. He also referred to a WWW 2006 paper he co-authored on extracting news-related queries from web query logs. The results of these efforts led to measurable improvements in click-based metrics of user happiness.

Ilya talked about a variety of efforts to support cross-lingual search. Russian users enter a significant fraction (about 15%) of non-Russian queries, but many still prefer Russian-language results. For example, a search for a company name return that company’s Russian-language home page if one is available. Yandex implements language personalization by learning a user’s language knowledge and using it as a factor in relevance computation. Yandex also uses machine translation to serve results for Russian-language queries when there are no relevant Russian-language results.

Ilya concluded by pitching the efforts that Yandex is making to participate in and support the broader information retrieval community, including running (and releasing data for) a relevance prediction challenge. It’s great to see a reminder that there is more to web search than Google vs. Bing, and refreshing to see how much Yandex shares its methodology and results with the IR community.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

Those of you who attended the SIGIR 2009 Industry Track had the opportunity to hear Yahoo researcher Vanja Josifovski make an eloquent case for ad retrieval as a new frontier of information retrieval. At the CIKM 2011 Industry Event, Vanja delivered an equally compelling presentation entitled “Toward Deep Understanding of User Behavior: A Biased View of a Practitioner“.

Vanja first offered a vision in which the web of the future will be  your life partner, delivering life-long pervasive personalized experience. Everything will be personalized, and the experience will pervade your entire online experience — from your laptop to your web-enabled toaster.

He then brought us back to the state of personalization today. For search personalization, the low entropy of query intent makes it difficult — or too risky — to significantly outperform the baseline of non-personalized search. In his view, the action today is in content recommendation and ad targeting, where there is high entropy of intent and lots of room for improvement over today’s crude techniques.

How do we achieve these improvements? We need more data, larger scale, and better methods for reasoning about data. In particular, Vanja noted the data we have today — searches, page views, connections, messages, purchases — represents the user’s thin observable state. In contrast, we lack data about the user’s internal state, e.g., is the user jet-lagged or worried about government debt. Vanja said that the only way to get more data is to motivate users by creating value for them with it — i.e., social is give to get.

Of course, we can’t talk about user’s hidden data without thinking about privacy. Vanja asserts that privacy is not dead, but that it’s in hibernation. So far, he argued, we’ve managed with a model of industry self-governance with relatively minor impact from data leaks — specifically as compared to the offline world. But he is apprehensive at the prospect of a major privacy breach inducing legislation that sets back personalization efforts for decades.

Vanja then talked about current personalization methods, including learning relationships among features, dimensionality reduction, and smoothing using external data. He argues that many of the models are mathematically very similar to one another, and it is difficult to analyze the relative merits of the models as opposed to other implementation details of the systems that use them.

Finally, Vanja touched on scale issues. He noted that the MapReduce framework imposes significant restrictions on algorithms used for personalization, and that we need the right abstractions for modeling in parallel environments.

ps. My thanks to Jeff Dalton for live-blogging his notes.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

Given the extraordinary ascent of all things social in today’s online world, we could hardly neglect this theme at the CIKM 2011 Industry Event. We were lucky to have Ed Chi, who recently left the PARC Augmented Social Cognition Group to work on Google+, presenting “Model-Driven Research in Social Computing“.

Ed warned us at the beginning of the talk that his focus would be on work he’d done prior to joining Google. Nonetheless, he offered an interesting collection of public statistics about social activity associated with Google properties: 360M words per day being published on Blogger, 150 years of YouTube video being watched everyday on Facebook, and 40M+ people using Google+. Regardless of how Google has fared in the competition for social networking mindshare, Google is clearly no stranger to online social behavior.

Ed then dove into recent research that he and colleagues have done on Twitter activity. Since all of the papers he discussed are available online, I will only touch on highlights. I encourage you to read the full papers:

• Want to be Retweeted? Large Scale Analytics on Factors Impacting Retweet in Twitter Network
• Tweets from Justin Bieber’s Heart: the Dynamics of the “Location” Field in User Profiles
• Is Twitter a Good Place for Asking Questions? A Characterization Study
• Language Matters in Twitter: A Large Scale Study
• Eddi: Interactive Topic-based Browsing of Social Status Streams
• Short and Tweet: Experiments on Recommending Content from Information Streams
• Speak Little and Well: Recommending Conversations in Online Social Streams

Ed talked at some length about language-dependent behavior on Twitter. For example, tweets in French are more likely to contain URLs than those in English, while tweets in Japanese are less likely (perhaps because the language is more compact relative to Twitter’s 140-character limit?). Tweets in Korean are far more likely to be conversational (i.e., explicitly mentioning or replying to other users) than those in English. These differences remind us to be cautious in generalizing our understanding of online social behavior from the behavior of English-speaking users. Ed also talked about cross-language “brokers” who tweet in multiple languages: he sees these as indicating connection strength between languages, as well as giving us insight to improve cross-­language communication.

Ed then talked about ways to reduce information overload in social streams. These included Eddi, a tool for summarizing social streams, and zerozero88, a closed experiment to produce a personal newspaper from a tweet stream. In analyzing the results of the zerozero88 experiment, Ed and his colleagues found that the most successful recommendation strategy combined users’ self-voting with social voting by their friends of friends. They also found that users wanted both relevance and serendipity — a challenge since the two criteria often compete with one another.

Ed concluded by offering the following design rule: since interaction costs determine number of the people who participate in social activity, get more people into the system by reducing interaction cost. He asserted that this is a key design principle for Google+.

My skepticism about Google’s social efforts is a matter of public record (cf. Social Utility, +/- 25%; Google±?). But hiring Ed Chi was a real coup for Google, and I’m optimistic about what he’ll bring to the Google+ effort.

ps. My thanks to Jeff Dalton for live-blogging his notes.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

One of the recurring themes at the CIKM 2011 Industry Event was that not all search is web search. Stephen Robertson, in advocating why recall matters, noted that web search was exceptional rather than typical as an information retrieval domain. Khalid Al-Kofahi on spoke about the challenges of legal search. Focusing on a different vertical, Funnelback Chief Scientist David Hawking spoke about “Search Problems and Solutions in Higher Education“.

David spent most of the presentation focusing on work that Funnelback did for the Australian National University. Funnelback was originally developed by CSIRO and the ANU under the name Panoptic.

The ANU has a substantial web presence, comprised of hundreds of sites and over a million pages. Like many large sites, it suffers from propagation delay: the most important pages are fresh, but material on the outposts can be stale. Moreover, there is broad diversity of authorship.

The university also has a strong editorial stance for ranking search results: the search engine needs to identify and favor official content. Given the proliferation of unofficial content, it can be a challenge to identify official sites based on signals like incoming link count, click counts, and the use of official style templates.

David described a particular application that Funnelback developed for ANU: a university course finder. The problem is similar to that of ecommerce search and calls for similar solutions, e.g., faceted search, auto-complete, and suggestions of related queries. And, just as in ecommerce, we can evaluate performance in terms of conversion rate.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

Large-scale computation was, not surprisingly, a major theme at the CIKM 2011 Industry Event. Ben Greene, Director of SAP Research Belfast, delivered a presentation on “Large Memory Computers for In-Memory Enterprise Applications“.

Ben started by defining in-memory computing as “technology that allows the processing of massive quantities of real time data in the main memory of the server to provide immediate results from analyses and transactions”. He then asked whether the cloud enables real-time computing, since there is a clear market hunger for cloud computing to solve the problems of our current enterprise systems.

Not surprisingly, he advocated in-memory computing as the solution for those problems. Like John Ousterhout and the RAMCloud team, he sees the need to scale DRAM memory independently from physical boxes. He proposed a model of coherent shared memory, using high-speed low-latency networks and separating the data transport and cache layers into a separate tier below the operating system. The goal: no server-side application caches, DRAM-like latency for physically distributed databases, and in fact no separation between the application server and the database server.

Ben argued that coherent shared memory can dramatically lower the cost of in-memory computing while minimizing the pain for application developers. He also offered some benchmarks for SAP’s BigIron system to demonstrate the performance improvements.

In short, Ben offered a vision of in-memory computing as a reincarnation of the mainframe. It was an interesting and provocative presentation, and my only regret is that we couldn’t stage a debate between him and Jeff Hammerbacher over the future of large-scale enterprise computing.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

I’m of course delighted that one of my colleagues at LinkedIn was able to participate in the CIKM 2011 Industry Event. Principal software engineer Chavdar Botev delivered a presentation on “Databus: A System for Timeline-Consistent Low-Latency Change Capture“.

LinkedIn processes a massive amount of member data and activity. It has over 135M members and is growing faster than two new members per second. Based on recent measurements, those members are on track to perform more than four billion searches on the LinkedIn platform in 2011. All of this activity requires a data change capture mechanism that allows external systems, such as its graph index and real-time full-text search index Zoie, to act as subscribers in user space and stay up to date with constantly changing data in the primary stores.

LinkedIn has built the Databus system to meet these needs. Databus meets four key requirements: timeline consistency, guaranteed delivery, low latency, and user-space visibility. For example, edits to member profile fields, such as companies and job titles, need to be standardized. Also, in order to give recruiters act quickly on feedback to their job postings, we need to be able to propagate the changes to the job description in near-real-time.

Databus propagates data changes throughout LinkedIn’s architecture. When there is a change in a primary store (e.g., member profiles or connections), the changes are buffered in the Databus Relay through a push or pull interface. The relay can also capture the transactional semantics of updates. Clients poll for changes in the relay. If a client falls behind the stream of change events in the relay, it is redirected to a Bootstrap database that delivers a compressed delta of the changes since the last event seen by the client.

In contrast to generic message systems (including the Kafka system that LinkedIn has open-sourced through Apache), Databus has moreinsight in the structure of the messages and can thus do better than just guaranteeing message-level integrity andtransactional semantics for communication sessions.

I tend to live a few levels above core infrastructure, but I’m grateful that Chavdar and his colleagues build the core platform that makes all of our large-scale data collection possible. After all, without data we have no data science.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

The original program for the CIKM 2011 Industry Event featured Peter Jackson, who was chief scientist at Thomson Reuters and author of numerous books and papers on natural language processing. Sadly, Peter died on August 3,2011. Thomson Reuters R&D VP of Research Khalid Al-Kofahi graciously agreed to speak in his place, delivering a presentation on  ”Combining Advanced Search Technology and Human Expertise in Legal Research“.

Khalid began by giving an “83-second” overview of the US legal system, laying out the roles of the law, the courts, and the legislature. He did so to provide the context for the domain that Thomson Reuters serves — namely, legal information. Legal information providers curate legal information, enhance it editorially and algorithmically, and work to make legal information findable and explainable in particular task contexts. He then worked through an example of how a case law document (specifically, Burger King v. Rudzewicz), appears in WestLawNext, with annotations that include headnotes, topic codes, citation data, and historical context.

Channelling William Goffman, Khalid asserted that a document’s content (words, phrases, metadata) are not sufficient to determine its aboutness and importance. Rather, we also have to consider what other people say about the document and how they interact with it. This is especially true in the legal domain because of the precedential nature of law. He then framed legal search in terms of information retrieval metrics, stating the requirements as completeness (recall), accuracy (precision), and authority. Not surprisingly, Khalid agreed with Stephen Robertson’s emphasis on the importance of recall.

Speaking more generally, Khalid noted that vertical search is not just about search. Rather, it’s about findability. which includes navigation, recommendations, clustering, faceted classification, collaboration, etc. Most importantly, it’s about satisfying a set of well-understood tasks. And, particularly in the legal domain, customers demand explainable models. Beyond this demand, explainability serves an additional purpose: it enables the human searcher to add value to the process (cf. human-computer information retrieval).

It is sad to lose a great researcher like Peter Jackson from our ranks, but I am grateful that Khalid was able to honor his memory by presenting their joint work at CIKM. If you’d like to learn more, I encourage you to read the publications on the Thomson Reuters Labs page.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

The third speaker in the program was Cloudera co-founder and Chief Scientist Jeff Hammerbacher. Jeff, recently hailed by Tim O’Reilly as one of the world’s most powerful data scientists, built the Facebook Data Team, which is most known for open-source contributions that include Hive and Cassandra. Jeff’s talk was entitled “Experiences Evolving a New Analytical Platform: What Works and What’s Missing“. I am thankful to Jeff Dalton for live-blogging a summary.

Jeff’s talk was a whirlwind tour through the philosophy and technology for delivering large-scale analytics (aka “big data”) to the world:

1) Philosophy

The true challenges in the task of data mining are creating a data set with the relevant and accurate information and determining the appropriate analysis techniques. While in the past it made sense to plan data storage and structure around the intended use of the data, the economics of storage and the availability of open-source analytics platforms argue for the reverse: data first, ask questions later; store first, establish structure later. The goal is to enable everyone — developers, analysts, business users — to “party on the data”, providing infrastructure that keeps them from clobbering one another or starving each other of resources.

2) Defining the Platform

No one just uses a relational database anymore. For example, consider Microsoft SQL Server. It is actually part of a unified suite that includes SharePoint for collaboration, PowerPivot for OLAP, StreamInsight for complex event processing (CEP), etc. As with the LAMP stack, there is a coherent framework analytical data management which we can call an analytical data platform.

3) Cloudera’s Platform

Cloudera starts with a substrate architecture of Open Compute commodity Linux servers configured using Puppet and Chef and coordinated using ZooKeeper. Naturally this entire stack is open-source. They use HFDS and Ceph to provide distributed, schema-less storage. They offer append-only table storage and metadata using Avro, RCFile, and HCatalog; and mutable table storage and metadata using HBase. For computation, they offer YARN (inter-job scheduling, like Grid Engine, for data intensive computing) and Mesos for cluster resource management; MapReduce, Hamster (MPI), Spark, Dryad / DryadLINQ, Pregel (Giraph), and Dremel as processing frameworks; and Crunch  (like Google’s FlumeJava), PigLatin, HiveQL, and Oozie as high-level interfaces. Finally, Cloudera offers tool access through FUSE, JDBC, and ODBC; and data ingest through Sqoop and Flume.

4) What’s Next?

For the substrate, we can expect support for fat servers with fat pipes, operating system support for isolation, and improved local filesystems (e.g., btrfs). Storage improvements will give us a unified file format, compression, better performance and availability, richer metadata, distributed snapshots, replication across data centers, native client access, and separation of namespace and block management. We will see stabilization of our existing compute tools and better variety, as well as improved fault tolerance, isolation and workload management, low-latency job scheduling, and a unified execution backend for workflow. And we will see better integration through REST API access to all platform components, better document ingest, maintenance of source catalog and provenance information, and an integration beyond ODBC with analytics tools. We will also see tools that facilitate that transition from unstructured to structured data (e.g. RecordBreaker).

Jeff’s talk was as information-dense as this post suggests, and I hope the mostly-academic CIKM audience was not too shell-shocked. It’s fantastic to see practitioners not only building essential tools for research in information and knowledge management, but reaching out to the research community to build bridges. I saw lots of intense conversation after his talk, and I hope the results realize the two-fold mission of the Industry Event, which is to give  researchers an opportunity to learn about the problems most relevant to industry practitioners, and to offer practitioners an opportunity to deepen their understanding of the field in which they are working.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

The second speaker in the program was Metaweb co-founder John Giannadrea. Google acquired Metaweb last year and has kept its promise to to maintain Freebase as a free and open database for the world (including for rival search engine Bing – though I’m not sure if Bing is still using Freebase). John’s talk was entitled “Freebase – A Rosetta Stone for Entities“. I am thankful to Jeff Dalton for live-blogging a summary.

John started by introducing Freebase as a representation of structured objects corresponding to real-world entities and connected by a directed graph of relationships. In other words, a semantic web. While it isn’t quite web-scale, Freebase is a large and growing knowledge base consisting of 25 million entities and 500 million connections — and doubling annually. The core concept in Freebase is a type, and an entity can have many types. For example, Arnold Schwarzenegger is a politician and an actor. John emphasized the messiness of the real world. For example, most actors are people, but what about the dog who played Lassie? It’s important to support exceptions.

The main technical challenge for Freebase is reconciliation — that is, determining how similar a set of data is to existing Freebase topics. John pointed out how critical it is for Freebase to avoid duplication of content, since the utility of Freebase depends on unique nodes in its graph corresponding to unique objects in the world. Freebase obtains many of its entities by reconciling large, open-source knowledge bases — including Wikipedia, WordNet, Library of Congress Authorities,  and metadata from the Stanford Library. Freebase uses a variety of tools to implement reconciliation, including Google Refine (formerly known as Freebase Gridworks) and Matchmaker, a tool for gathering human judgments. While reconciliation is a hard technical problem, it is made possible by making inferences across the web of relationships that link entities to one another.

John then presented Freebase as a Rosetta Stone for entities on the web. Since an entity is simply a collection of keys (one of which is its name), Freebase’s job is to reverse engineer the key-value store that is distributed among the entity’s web references, e.g., the structured databases backing web sites and encoding keys in URL parameters. He noted that Freebase itself is schema-less (it is a graph database), and that even the concept of a type is itself an entity (“Type type is the only type that is an instance of itself”). Google makes Freebase available through an API and the Metaweb Query Language (MQL).

Freebase does have its challenges. The requirement to keep out duplicates is an onerous one, as they discovered when importing a portion of the Open Library catalog. Maintaining quality calls for significant manual curation, and quality varies across the knowledge base. John asserted that Freebase provides 99% accuracy at the 95th percentile, though it’s not clear to me what that means (update: see Bill’s comment below).

While I still have concerns about Freebase’s robustness as a structured knowledge base (see my post on “In Search Of Structure“), I’m excited to see Google investing in structured representations of knowledge. To hear more about Google’s efforts in this space, check out the Strata New York panel I moderated on Entities, Relationships, and Semantics — the panelists included Andrew Hogue, who leads Google’s structured data and information extraction group and managed me during my year at Google New York.

Earlier this year, I had the privilege to moderate a panel at Strata New York 2011 on Entities, Relationships, and Semantics: the State of Structured Search. The four panelists are people I’ve had the pleasure to work with over the years: Andrew Hogue (Google), Breck Baldwin (Alias-i), Evan Sandhaus (New York Times), Wlodek Zadrozny (IBM Research). They work on some of the world’s largest structured search problems — from offering users structured search on Google’s web corpus to building a computing system that defeated Jeopardy! champions in an extreme test of natural language understanding.

O’Reilly has compiled the nearly 50 hours of video from the conference and made the collection available for purchase. I was lucky to attend all of the keynotes and many of the breakout sessions, and I highly recommend them. In the meantime, you can see a recording of the panel I moderated.

Dan Woods has been interviewing a variety of folks to answer the question: “What is a data scientist?“, and I had the honor to participate in his series.

Here is a teaser of my interview:

Above all, a data scientist needs to be able to derive robust conclusions from data. But a data scientist also needs to possess creativity and strong communication skills. Creativity drives the process of hypothesis generation, i.e., picking the right problems to solve that will create value for users and drive business decisions.

Read the rest on Forbes.com. And thanks to Drew Conway for the awesome data science Venn diagram above.

Last week, I had the privilege to present a tutorial at the 5th ACM International Conference on Recommender Systems (RecSys 2011). Given my passion for HCIR and my advocacy for transparency in recommender systems, it shouldn’t surprise regular readers that I focused on both. Unfortunately the tutorial was not recorded, but I hope the slides above prove useful. I also encourage you to take a look at the other tutorials, whose slides are posted on the conference site.

Wednesday evening before the workshop, we were treated to a welcome reception, the first of a few meals provided by Google’s excellent chefs. It was a great opportunity to reconnect with old friends and meet many first-time HCIR attendees.

Thursday started with a scrumptious breakfast that included chilaquiles, coconut fritters, and bacon. Last year’s keynote and this year’s local host Dan Russell pulled all the stops — apparently BigTable is the only Google cafe that serves bacon for breakfast! We then proceeded to a poster boaster session in which each poster presenter had a minute to pitch his or her poster. This session set the tone for the rest of the workshop: concentrated ideas and intense audience engagement.

Then came this year’s keynote, Gary Marchionini. It was a particular treat to have Gary as a keynote, since his lecture on “Toward Human-Computer Information Retrieval” inspired me to conceive the HCIR workshop back in 2007. And Gary delivered the goods. He started with a review of the history of HCIR, including some lesser known figures like Don Hawkins (who was in the audience) , Pauline Cochrane, Richard Marcus, and Charles Meadow.  He brought a few chuckles by citing Nick Belkin (who was present) and Sue Dumais (who was not) as the father and mother of HCIR. Naturally he described some of his own work at the University of North Carolina, including the Open Video, Relation Browser, and ResultsSpace projects.But the highlight of his talk was a graph he presented showing two paths to the same user end-state, one of the paths being a smooth progression and the other being a roller-coaster of ups and down. The question of which one was better drew a wide variety of responses, my favorite being Gene Golovchinsky observing that learning is the friction of the information-seeking process.

We broke for coffee and then came back to the first session of paper presentations. Sofia Athenikos presented a semantic search engine that outperformed IMDB in a user study. Chang Liu explored the effect of task difficulty and domain knowledge on dwell times, finding counterintuitive results (at least for me) regarding the correlation of expertise to dwell time. Jingjing Liu presented research on knowledge examination in multi-session tasks. Then came the lightning talks: Mark Smucker on how users examine and process ranked document lists; Jin Kim on simulating associative browsing; Bill Kules on visualizing the stages of exploratory search; and Michael Cole on user domain knowledge and eye movement patterns during search. Way too much goodness to summarize here — I suggest you read the full papers on the workshop site.

Then came lunch — again in BigTable, but this time with outdoor seating — and the poster session. As always, this it the most interactive part of the day: two hours of non-stop discussion that start over food and end with prying people away from discussions about posters. I was especially proud of LinkedIn’s contributions to the poster session, which covered faceted search log analysis, social navigation, and whether it is time to abandon abandonment.

Then back to the second session of  paper presentations. Luanne Freund talked about document usefulness and genre, finding that genre, besides being hard for users to reliably identify, only matters for tasks that involve doing, deciding, learning; but not for those that involve fact finding or problem solving. Gene Golovchinsky presented work on designing for collaboration in information seeking, previewing the system he used for his challenge entry.  Alyona Medelyan used the Pingar search engine to evaluate how search interface features affect performance on biosciences tasks. Then more lightning talks: Rob Capra analyzing faceted search on mobile devices; Keith Bagley on conceptual mile markers for exploratory search; Xiaojun Yuan on how cognitive styles affect user performance; and Mike Zarro on using social tags and controlled vocabularies as search filters.

Last but not least came the HCIR Challenge:

The HCIR 2011 Challenge focuses on the case where recall is everything – namely, the problem of information availability. The information availability problem arises when the seeker faces uncertainty as to whether the information of interest is available at all. Instances of this problem include some of the highest-value information tasks, such as those facing national security and legal/patent professionals, who might spend hours or days searching to determine whether the desired information exists.

The corpus we will use for the HCIR 2011 Challenge is the CiteSeer digital library of scientific literature. The CiteSeer corpus contains over 750,000 documents and provides rich meta-data about documents, authors, and citations.

There were four entries:

• FreeSearch – Literature Search in a Natural Way
  Claudiu S. Firan, Wolfgang Nejdl, Mihai Georgescu (University of Hanover), and Xinyun Sun (DEKE Lab MOE, Renmin)
  
• Session-based search with Querium
  Gene Golovchinsky (FX Palo Alto Lab) and Abdigani Diriye (University College London)
  
• GisterPro
  David L.Ostby and Edmond Brian (Visual Purple)
  
• Query Analytics Workbench
  Antony Scerri, Matthew Corkum, Keith Gutfreund, Ron Daniel Jr., Michael Taylor (Elsevier Labs)

The competition was fierce. Claudiu showed off the Faceted DBLP interface, which is well suited to the information availability task on CiteSeer data. Ed showed how GisterPro uses visualization to support the information seeking process. But it came down to a close call between the Query Analytics Workbench and Querium. Despite the Elsevier team’s impressive functionality and animated presentation, Gene’s simpler interface and application of ranked fusion won the day. Congratulations to Gene and Abdigani, this year’s HCIR Challenge winners!

We wrapped up the evening at the Tied House, a local microbrewery. And of course the discussion turned to where, when, and how we will hold next year’s workshop. Watch this space. In the meantime, my heartfelt thanks to everyone who made this year’s workshop such a success — and especially to our sponsors. Thank you Endeca, Kent State, Microsoft, and Google!

Today is a wonderful day for Endeca and Oracle! Oracle has announced that it has entered into an agreement to acquire Endeca, bringing together two of the powerhouses of information access. Quoting from the announcement: “The combination of Oracle and Endeca is expected to create a comprehensive technology platform to process, store, manage, search and analyze structured and unstructured information together. ”

As part of Endeca’s founding team, I am very proud to see this day. My ten years at Endeca were a formative experience that established my professional identity and inspired my passion to pursue the vision of human-computer information retrieval (by happy coincidence, the 5th annual HCIR workshop take place on Thursday). Reading Oracle’s presentation about the acquisition, I’m excited to see how Endeca’s technology will play a key role in unifying structured and unstructured data management and analysis for Oracle’s customers.

I take pride in my contributions to Endeca — I still slip sometimes and refer to Endeca as “we”. But the real heroes here are the folks — and especially the leadership —  who have seen this journey through from start to finish. In particular, I am grateful to Steve Papa, Pete Bell, Adam Ferrari, Jack Walter, Keith Johnson, Nik Bates-Haus, and Jason Purcell for everything they have done to bring about this extraordinary outcome.

Finally, excited as I am about this event, it is only the beginning. I am excited to see Endeca’s people and technology powering one of the world’s largest enterprise software companies. Looking forward to the next play!

I had a great audience, including the department head! Of course that meant fielding tough questions, but that’s what makes it fun to present at my alma mater. Now that it’s been over a decade since my defense, I can handle the tough questions.

Unfortunately there is no video, but hopefully the slides are reasonably self-explanatory. If you have questions, please ask them in the comments.

Tonight I’m taking a red-eye to Pittsburgh so that I can spend three days at my (doctoral) alma mater, CMU. In addition to spending time with lots of great students and faculty, my goal is to communicate a taste of the hard computer science problems we are solving (or trying to solve!) at LinkedIn. I’m giving a tech talk Tuesday afternoon, joining my colleagues for an info session Tuesday evening, and participating in the Technical Opportunities Conference (TOC) Wednesday.

Here’s a teaser for my tech talk:

You can find more details about LinkedIn’s visits to CMU and other campuses at http://studentcareers.linkedin.com/.

Hopefully some of you are attending the O’Reilly Strata Conference in New York this Thursday and Friday. If so, I encourage you to attend my panel session on “Entities, Relationships, and Semantics: the State of Structured Search“:

Structured search improves the search experience through the identification of entities and their relationships in documents and queries. This panel will explore the current state of structured and semi-structured search, as well as exploring the open problems in an area that promises to revolutionize information seeking.

The four panelists work on some of the world’s largest structured search problems, from offering users structured search on Google’s web corpus to building a computing system that defeated Jeopardy! champions in an extreme test of natural language understanding. They work on the data, tools, and research that are driving this field. They are all excellent researchers and presenters, promising to offer a informative and engaging panel discussion, for which I will act as moderator.

Panelists:

• Andrew Hogue is a Senior Staff Engineer and Engineering Manager in the Search Quality group at Google New York. He has worked on a wide array of projects including question answering, Google Squared, sentiment analysis, local and product search, and Google Goggles. His is interested in the areas of structured data, information extraction, and machine learning, and their applications to search and search interfaces. Prior to Google, he earned a M.Eng. and B.S. in Computer Science from MIT.

• Breck Baldwin is the President of Alias-i, creators of the popular LingPipe computational linguistics toolkit. He received his Ph.D. in computer science in 1995 from the University of Pennsylvania. In the time between his thesis on coreference resolution and evaluation and founding Alias-i in 1999, Breck worked on DARPA-funded projects through the University of Pennsylvania.

• Evan Sandhaus works as the Semantic Technologist in The New York Times Research and Development Labs. He is spearheading The New York Times Linked Open Data Strategy and overseeing the release of 1.8 million documents to the computer science research community. Previously, Evan helped to put The New York Times on Google Earth, collaborated with New York University to explore new directions in News Search, and worked to bring The New York Times to Facebook.

• Wlodek Zadrozny is an IBM Researcher working on natural language applications. Most recently he worked on text sources for Watson (IBM’s Jeopardy chamption) and applying related DeepQA technology to business problems. His previous work ranged from language processing research to product development and technical planning; in particular, he lead the development of interactions systems that used speech, natural language and focused search. Wlodek Zadrozny received a Ph.D. in Mathematics, from the Polish Academy of Science.

And one more thing. Karaoke at Second on Second in the East Village on Friday night. It’s an unofficial Strata after-party, so come join us Big Data folks for some Big Fun.

I grew up in New York City. On September 11th, 2001, I was in Cambridge, Massachusetts, desperately trying to get through to my parents by all means of communication at my disposal. My dad worked at 40 Worth Street, only a few blocks away from the World Trade Center. Thankfully none of my family or friends were harmed that day, but that fateful event ten years ago left a mark on the world that no one of my generation will ever forget.

Fortunately I have happier associations with this anniversary.

On September 11th, 1999, I boarded an Amtrak from New York to Boston to join Steve Papa, Pete Bell, Dave Gourley, Fritz Knabe, Jack Walter, and Phil Braden to start the company that would eventually be named Endeca. I had no way of knowing whether we would persuade VCs to fund us beyond our six months of seed investment, let alone that we would develop a technology that to revolutionize the search experience of millions of users around the world. Our modest ambition was to build a better way to find stuff on eBay. That goal remains unfulfilled, but 44 of the top 100 online retailers use Endeca, which isn’t too shabby. Especially considering that Endeca has expanded well beyond online retail into domains like manufacturing, business intelligence, and government.

On Seprtember 11th, 2002, I gathered the Endeca founding team for a dinner to celebrate the company’s 3rd birthday. Given my reputation for general irreverence, I feared that my colleagues would think this was a stunt to mock the memory of the more familiar 9/11. But it was quite the opposite. September 11th, 1999 was a turning point in my professional life, and no terrorist was going to take that happiness away from me. To this day I am grateful that my colleagues recognized my sincerity and joined me in this celebration.

The dinner that night was an emotional one: 2002 had been a tough year for the software industry — one in which we saw many of our peer companies fold. Fortunately it was the beginning of much better times for us: from 2003 to 2006, Endeca was the fastest growing private company in Massachusetts. No IPO yet, but the rumors are encouraging.

I left Endeca almost two years ago, going to Google and then LinkedIn. But I will always have fond memories of the decade I spent at Endeca — an experience that established much of the passion that drives me today. I am very proud to have been part of the founding team of such a great company, even if now I can only follow from a distance.

Happy birthday, Endeca, and many more to come!

As many of you know, I’m a proud alumnus of the CMU School of Computer Science (yes, I also attended the CMU of Massachusetts). I’m delighted to have the opportunity to spend a few days on campus this month, and I hope that I’ll have a chance to meet with lots of students and faculty while I’m there.

Specifically, I’ll be giving a talk at Eugene Fink’s Intelligence Seminar on Tuesday, September 20th at 3:30pm in Gates-Hillman 4303:

Keeping It Professional: Relevance, Recommendations, and Reputation at LinkedIn

LinkedIn operates the world’s largest professional network on the Internet with more than 120 million members in over 200 countries. In order to connect its users to the people, opportunities, and content that best advance their careers, LinkedIn has developed a variety of algorithms that surface relevant content, offer personalized recommendations, and establish topic-sensitive reputation — all at a massive scale. In this talk, I will discuss some of the most challenging technical problems we face at LinkedIn, and the approaches we are taking to address them.

I hope to see all of you there! My colleagues and I will also be hosting an information session that same Tuesday at 6pm in Porter Hall, Room 125B, as well as participating in the Technical Opportunities Conference Tuesday and Wednesday. And of course LinkedIn will be conducting on-campus interviews: those will take place all day on Thursday, September 22nd.

If you are a CMU student interested in opportunities at LinkedIn, please apply through TartanTrak (yes, I wish you could just apply with LinkedIn – we’ll get there!). Of course, feel free to reach out to me personally at dtunkelang@linkedin.com. We already have more applicants than slots, but I promise that every application will be considered. I’m very excited to recruit CMU students to strengthen our growing team of software engineers and data scientists.

See you soon, and let’s go Tartans!

A few days ago, our CEO Jeff Weiner led a session at LinkedIn on how to “close” candidates — that is, how to persuade candidates to join your team once you have found and interviewed them. Since not everyone has the opportunity to work at LinkedIn and experience Jeff’s leadership first-hand, I thought I’d share some of his wisdom here.

The key take-away  was that closing a candidate is not about selling the job or company to the candidate, but rather working with the candidate to figure out what the candidate wants and whether the job will help him or her achieve that desire. As an employer, you need to do three things to close a candidate:

1) Figure out what is the candidate’s dream.
2) Determine if job and candidate are the right fit.
3) Communicate your own passion.

Let’s take these one at a time.

Dream.

As I’ve written here in the past, we have to dare to dream. Most of us rely on jobs to sustain us and our loved ones — and for some a job is nothing more than that. There’s no shame in having a dream that is unrelated to a job — Franz Kafka famously worked in a variety of “bread jobs” in order to pay the bills while he wrote novels. Others find their calling as humanitarians, activists, or care givers. It’s easy for many of us to forget that life isn’t always about work.

But the great thing about working in technology is that you can get paid to fulfill your own dream. Look at Larry and Sergey, who set out to organize the world’s information. Or Steve Jobs, whose dream has been to create innovative products. Not everyone is as specific in their dreams or as successful in realizing them, but, as the saying goes, you have to be in it to win it.

Convincing a person to accept a job offer works best when that job brings the person closer to fulfilling his or her dream. My own decisions to go to Google and then LinkedIn are good examples. Working at Endeca drove me to pursue a vision of HCIR — to optimize the way people and machines work together to solve information seeking and exploration tasks. At Google, I hoped to bring exploratory search to the open web. I’ll concede that I did not make much headway, but I’m glad that I tried.

And at LinkedIn, I work on problems that not only stretch the boundaries of information science, but whose solutions help millions of other people achieve their dreams by making them more successful professionally. My dream is to truly reduce HCIR to practice so that people can lead better and more productive lives. Once the folks at LinkedIn understood my dream, closing me was just a matter of offering me the keys to make that dream a reality.

If you want someone to work at your company, get to know that person’s dreams. If the job you are offering can’t help him or her realize those dreams, be honest about it. It’s better for both of you, and for a world that is better off with people devoting their lives’ work to fulfilling their dreams.

Fit.

Fit is a two way street: the candidate should be right for the job, and the job should be right for the candidate. The interviewing process typically focuses on establishing the former, but we often forget that the candidate’s decision focuses on the latter. Just because someone is capable of doing a job doesn’t mean it’s the right job for that person.

For me, fit means many things. A work environment where people work hard and take the company’s success personally. Incentives that allow everyone to win, rather than a zero-sum game where people compete for scarce opportunities. Openness, since I’m someone who lives most of my life in public. I could go on — but I hope you get the general idea. Fit is the set of functional and non-functional requirements that determine whether someone will enjoy a job. And people who enjoy their jobs tend to be productive and stay a while.

If you are trying to persuade someone to accept a job offer, you have to see the decision from that person’s point of view. In other words, ask yourself — and convincingly answer — why the job is the right fit for the candidate. That means accepting the possibility that is isn’t the right fit, and doing right by the candidate even if that means backing off.

Passion.

Choosing a job is one of the most important life decisions that people make. It’s not quite up there with getting married or having a child, but it’s a a decision that most people take (and should take) very seriously. Some people create spreadsheets of the pros and cons to compare opportunities and try to frame their decision as an optimization problem. Others go with their gut.

Those who know me personally — whether from face-to-face or online interaction — know that I wear my passion on my sleeve. I can’t understand how someone could get up in the morning and go to work without being passionate about his or her job. I know that many people don’t have a choice in the matter, and I pity them. In a country where most people take subsistence for granted, having a job you love strikes me as a necessity, rather than a luxury.

But what is clear is that if you, as an employer, are not passionate about what you do, you have no business expecting a candidate to take such a big leap of faith with you. Moreover, passion is hard to fake. As it should be — I’m not suggesting that employers should pretend to be excited about their jobs. Rather, your own sincere excitement is a baseline for those you hope to attract to your team. Passion is contagious, and passion is the raw material for making dreams come true.

Dream. Fit. Passion.

There you have it: dream, fit, passion. And remember, closing isn’t selling. Do right by the people you try to hire. After all, jobs are short, but careers are long. Celebrate everyone’s professional success, and take your losses in stride. I can tell you from experience that it all works out for the best.

Interviewing software engineers is hard. Jeff Atwood bemoans how difficult it is to find candidates who can write code. The tech press sporadically publishes “best” interview questions that make me cringe — though I love the IKEA question. Startups like Codility and Interview Street see this challenge as an opportunity, offering hiring managers the prospect of outsourcing their coding interviews. Meanwhile, Diego Basch and others are urging us to stop subjecting candidates to whiteboard coding exercises.

I don’t have a silver bullet to offer. I agree that IQ tests and gotcha questions are a terrible way to assess software engineering candidates. At best, they test only one desirable attribute; at worst, they are a crapshoot as to whether a candidate has seen a similar problem or stumbles into the key insight. Coding questions are a much better tool for assessing people whose day job will be coding, but conventional interviews — whether by phone or in person — are a suboptimal way to test coding strength. Also, it’s not clear whether a coding question should assess problem-solving, pure translation of a solution into working code, or both.

In the face of all of these challenges, I came up with an interview problem that has served me and others well for a few years at Endeca, Google, and LinkedIn. It is with a heavy heart that I retire it, for reasons I’ll discuss at the end of the post. But first let me describe the problem and explain why it has been so effective.

The Problem

I call it the “word break” problem and describe it as follows:

Given an input string and a dictionary of words,
segment the input string into a space-separated
sequence of dictionary words if possible. For
example, if the input string is "applepie" and
dictionary contains a standard set of English words,
then we would return the string "apple pie" as output.

Note that I’ve deliberately left some aspects of this problem vague or underspecified, giving the candidate an opportunity to flesh them out. Here are examples of questions a candidate might ask, and how I would answer them:

Q: What if the input string is already a word in the
   dictionary?
A: A single word is a special case of a space-separated
   sequence of words.

Q: Should I only consider segmentations into two words?
A: No, but start with that case if it's easier.

Q: What if the input string cannot be segmented into a
   sequence of words in the dictionary?
A: Then return null or something equivalent.

Q: What about stemming, spelling correction, etc.?
A: Just segment the exact input string into a sequence
   of exact words in the dictionary.

Q: What if there are multiple valid segmentations?
A: Just return any valid segmentation if there is one.

Q: I'm thinking of implementing the dictionary as a
   trie, suffix tree, Fibonacci heap, ...
A: You don't need to implement the dictionary. Just
   assume access to a reasonable implementation.

Q: What operations does the dictionary support?
A: Exact string lookup. That's all you need.

Q: How big is the dictionary?
A: Assume it's much bigger than the input string,
   but that it fits in memory.

Seeing how a candidate negotiates these details is instructive: it offers you a sense of the candidate’s communication skills and attention to detail, not to mention the candidate’s basic understanding of data structures and algorithms.

A FizzBuzz Solution

Enough with the problem specification and on to the solution. Some candidates start with the simplified version of the problem that only considers segmentations into two words. I consider this a FizzBuzz problem, and I expect any competent software engineer to produce the equivalent of the following in their programming language of choice. I’ll use Java in my example solutions.

String SegmentString(String input, Set<String> dict) {
  int len = input.length();
  for (int i = 1; i < len; i++) {
    String prefix = input.substring(0, i);
    if (dict.contains(prefix)) {
      String suffix = input.substring(i, len);
      if (dict.contains(suffix)) {
        return prefix + " " + suffix;
      }
    }
  }
  return null;
}

I have interviewed candidates who could not produce the above — including candidates who had passed a technical phone screen at Google. As Jeff Atwood says, FizzBuzz problems are a great way to keep interviewers from wasting their time interviewing programmers who can’t program.

A General Solution

Of course, the more interesting problem is the general case, where the input string may be segmented into any number of dictionary words. There are a number of ways to approach this problem, but the most straightforward is recursive backtracking. Here is a typical solution that builds on the previous one:

String SegmentString(String input, Set<String> dict) {
  if (dict.contains(input)) return input;
  int len = input.length();
  for (int i = 1; i < len; i++) {
    String prefix = input.substring(0, i);
    if (dict.contains(prefix)) {
      String suffix = input.substring(i, len);
      String segSuffix = SegmentString(suffix, dict);
      if (segSuffix != null) {
        return prefix + " " + segSuffix;
      }
    }
  }
  return null;
}

Many candidates for software engineering positions cannot come up with the above or an equivalent (e.g., a solution that uses an explicit stack) in half an hour. I’m sure that many of them are competent and productive. But I would not hire them to work on information retrieval or machine learning problems, especially at a company that delivers search functionality on a massive scale.

Analyzing the Running Time

But wait, there’s more! When a candidate does arrive at a solution like the above, I ask for an big O analysis of its worst-case running time as a function of n, the length of the input string. I’ve heard candidates respond with everything from O(n) to O(n!).

I typically offer the following hint:

Consider a pathological dictionary containing the words
"a", "aa", "aaa", ..., i.e., words composed solely of
the letter 'a'. What happens when the input string is a
sequence of n-1 'a's followed by a 'b'?

Hopefully the candidate can figure out that the recursive backtracking solution will explore every possible segmentation of this input string, which reduces the analysis to determine the number of possible segmentations. I leave it as an exercise to the reader (with this hint) to determine that this number is O(2ⁿ).

An Efficient Solution

If a candidate gets this far, I ask if it is possible to do better than O(2ⁿ). Most candidates realize this is a loaded question, and strong ones recognize the opportunity to apply dynamic programming or memoization. Here is a solution using memoization:

Map<String, String> memoized;

String SegmentString(String input, Set<String> dict) {
  if (dict.contains(input)) return input;
  if (memoized.containsKey(input) {
    return memoized.get(input);
  }
  int len = input.length();
  for (int i = 1; i < len; i++) {
    String prefix = input.substring(0, i);
    if (dict.contains(prefix)) {
      String suffix = input.substring(i, len);
      String segSuffix = SegmentString(suffix, dict);
      if (segSuffix != null) {
        memoized.put(input, prefix + " " + segSuffix);
        return prefix + " " + segSuffix;
      }
    }
  memoized.put(input, null);
  return null;
}

Again the candidate should be able to perform the worst-case analysis. The key insight is that SegmentString is only called on suffixes of the original input string, and that there are only O(n) suffixes. I leave as an exercise to the reader to determine that the worst-case running time of the memoized solution above is O(n²), assuming that the substring operation only requires constant time (a discussion which itself makes for an interesting tangent).

Why I Love This Problem

There are lots of reasons I love this problem. I’ll enumerate a few:

• It is a real problem that came up in the couse of developing production software. I developed Endeca’s original implementation for rewriting search queries, and this problem came up in the context of spelling correction and thesaurus expansion.
• It does not require any specialized knowledge — just strings, sets, maps, recursion, and a simple application of dynamic programming / memoization. Basics that are covered in a first- or second-year undergraduate course in computer science.
• The code is non-trivial but compact enough to use under the tight conditions of a 45-minute interview, whether in person or over the phone using a tool like Collabedit.
• The problem is challenging, but it isn’t a gotcha problem. Rather, it requires a methodical analysis of the problem and the application of basic computer science tools.
• The candidate’s performance on the problem isn’t binary. The worst candidates don’t even manage to implement the fizzbuzz solution in 45 minutes. The best implement a memoized solution in 10 minutes, allowing you to make the problem even more interesting, e.g., asking how they would handle a dictionary too large to fit in main memory. Most candidates perform somewhere in the middle.

Happy Retirement

Unfortunately, all good things come to an end. I recently discovered that a candidate posted this problem on Glassdoor. The solution posted there hardly goes into the level of detail I’ve provided in this post, but I decided that a problem this good deserved to retire in style.

It’s hard to come up with good interview problems, and it’s also hard to keep secrets. The secret may be to keep fewer secrets. An ideal interview question is one for which advance knowledge has limited value. I’m working with my colleagues on such an approach. Naturally, I’ll share more if and when we deploy it.

In the mean time, I hope that everyone who experienced the word break problem appreciated it as a worthy test of their skills. No problem is perfect, nor can performance on a single interview question ever be a perfect predictor of how well a candidate will perform as an engineer. Still, this one was pretty good, and I know that a bunch of us will miss it.

There are four outstanding information retrieval conferences coming up, and I will have the pleasure of participating in three of them. I’d like to make sure readers here are aware of all of them.

The first is KDD 2011, which will take place August 21-24, 2011 in San Diego, CA. The annual ACM SIGKDD conference is the premier international forum for data mining researchers and practitioners from academia, industry, and government to share their ideas, research results and experiences. KDD-2011 will feature keynote presentations, oral paper presentations, poster sessions, workshops, tutorials, panels, exhibits, demonstrations, and the KDD Cup competition. KDD-2011 will run from August 21-24 in San Diego, CA and will feature hundreds of practitioners and academic data miners converging on the one location.

I will not be attending KDD myself, but several of my colleagues will be there. In particular, Ron Bekkerman will be presenting a paper on “High-Precision Phrase-Based Document Classification on a Modern Scale”, as well as offering a tutorial on “Scaling Up Machine Learning: Parallel and Distributed Approaches”.

Workshop on Human-Computer Interaction and Information Retrieval (HCIR 2011) - Mountain View, CA – October 20, 2011

The second is HCIR 2011, the fifth annual HCIR workshop, which I am co-organizing. It will be held all day on Thursday, October 20th, 2011 at Google’s main campus in Mountain View, California. There will be a reception on Wednesday evening before the workshop. Our keynote speaker this year will be Gary Marchionini, Dean of the School of Information and Library Science, University of North Carolina at Chapel Hill. We are also excited to continue the HCIR Challenge, this year focusing on the problem of information availability, where the seeker faces uncertainty as to whether the information of interest is available at all. The corpus will be the CiteSeer digital library of scientific literature, which contains over 750,000 documents and provides rich meta-data about documents, authors, and citations.

Thanks to generous contributions made by Google, Microsoft Research, and Endeca, there will be no registration fee for HCIR this year. Information about how to register will be sent to authors of accepted position papers, research papers, and challenge reports. Note that the submission deadline has been extended by two weeks to Sunday, August 14th. I strongly encourage you to submit in one of these categories in you are working in this field.

The third is RecSys 2011, the 5th ACM International Conference on Recommender Systems. RecSys 2011 builds on the success of the Recommenders 06 Summer School in Bilbao, Spain and the series of four successful conference events from 2007 to 2010 in Minneapolis (2007), Lausanne (2008), New York (2009) and Barcelona (2010). In these events many members of the practitioner and research communities valued the rich exchange of ideas made possible by the shared plenary sessions. The 5th International conference will promote the same close interaction among practitioners and researchers.

I will be giving a tutorial at RecSys 2011 on “Recommendations as a Conversation with the User”.

The fourth is CIKM 2011, the 20th ACM Conference on Information and Knowledge Management. It will take place in Glasgow, Scotland, UK, 24th-28th October 2011. Since 1992, the CIKM has successfully brought together leading researchers and developers from the database, information retrieval, and knowledge management communities. The purpose of the conference is to identify challenging problems facing the development of future knowledge and information systems, and to shape future research directions through the publication of high quality, applied and theoretical research findings. CIKM 2011 will continue the tradition of promoting collaboration among multiple areas in the general areas of databases, information retrieval, and knowledge management.

I am proud to be organizing the CIKM 2011 Industry Event, which will feature such industry heavyweights as Stephen Robertson (Microsoft Research), John Giannandrea (Google), Vanja Josifovski (Yahoo! Research), Ilya Segalovich (Yandex), Jeff Hammerbacher (Cloudera), and Chavdar Botev (LinkedIn).

I’m very excited about all four of these opportunities to exchange ideas about information retrieval and related areas, and I am grateful to LinkedIn for supporting my participation, as well as that of my colleagues. I hope to see some of you at these events!

A major feature of the recently released Google+ is Circles, which allows you to “share relevant content with the right people, and follow content posted by people you find interesting.”

Most people seem to look at Circles as a privacy feature — and indeed Google’s official description gives the impression that Circles exist to manage privacy based on real-life social contexts. Of course, re-sharing can result in unintended consequences, and Google even offers a warning that:

Unless you disable reshares, anything you share (either publicly or with your circles) can be reshared beyond the original people you shared the content with. This could happen either through reshares or through mentions in comments.

Privacy is a big deal, especially for Google — and particularly in the context of rolling out a new social network. Still, I’m not persuaded that privacy is the only or even the primary concern motivating the concept of social circles.

Sharing content with someone is not just about giving that person permission to see it. Sharing content with someone asserts a claim on that person’s attention. While it may be a privilege for me to have access to your content, it may be even more of a privilege for you that I allocate my scarce attention to consume it.

What if we focus on routing content to the people who would find it most interesting? Such an approach works best if all of the shared content is public with respect to permissions — that is, people post it without any expectation of privacy. Twitter demonstrates that many people are comfortable with such a sharing model. Imagine if they could learn to trust a system that optimizes (or at least attempts to optimize) the allocation of everyone’s attention. This is not an easy problem by any means, nor is it one that is likely to be solved by algorithms alone. It will take a strong dose of HCIR to get it right. But, at least in my view, optimizing the allocation of human attention is the grand challenge that everyone working with information retrieval or social networks should be striving to address.

Privacy is important, and social networks should offer simple, robust privacy controls that users understand. We all have experienced the problem of filter failure. But sharing isn’t just about privacy. Our attention is our most precious cognitive asset, both as individuals and as a society, Moreover, our attention faces ever-increasing demands as our social lives evolve in an online world relatively free of physical constraints. Social network developers would do well to pay attention…to attention.

Diego Basch is the CEO and founder of IndexTank, a hosted search service that powers major web sites such as Reddit, Twitvid, blip.tv, as well as providing a WordPress plug-in for blogs (like this one). Diego gained his search experience working with Inktomi, where he wrote some of the world’s first web-scale link analysis algorithms. He is on a mission to make every search box blazing fast and useful.

So much brainpower is spent solving the wrong problems. The world is filled with solutions looking for problems that nobody has — as illustrated by a Google query for [stupidest inventions ever]. More often, people focus narrowly on a particular approach when they should focus on the problem the approach is intended to solve. Or they take a solution for one problem and assume it will apply to another.

Consider the emphasis that search engine developers place on relevance ranking. It is not hard to understand why web-scale search engines emphasize relevance. For example, a search on Google for [emergency locksmith] returns tens of billions of web pages, among which there are only a handful results that you want. Google must filter out the growing number of lead generation companies that spend a ton of money trying to game its results.

Most web and application developers are familiar with the concept of relevance, so they naturally assume that it should be the primary concern when they add search to their own sites or apps. When I talk to people who want full-text search for their 40,000 book titles or 100k classified ads, they ask me about all the ways they can tune relevance. But often they are focusing on a solution, rather than their fundamental problem.

Developers are (or should be!) trying to improve the user experience of their application search. Too often they wrongly assume that relevance is the single most important factor for optimizing this user experience. Let’s surface this confusion in a concrete example.

As a rock climber, once in a while I feel the aches and pains caused by the sport. As the years go by it’s very important to keep your tendons healthy if you do not want to take forced breaks (or type with one hand!). Rockclimbing.com is one of the most popular climbing sites, and I know some medical professionals who occasionally answer health-related questions there. Let’s search there for [tendon injury prevention].

In the above example, part of the problem is that the search results do not have contextual snippets. Maybe there is relevant information hiding behind a click, but the user has no way of knowing. More generally, there’s no hint as to what results could be better. Information such as score of the answer (which is available), the author’s bio (e.g. “climber, physical therapist”) would make the decision easier. If you need to click and scroll, search within the page, go back and try something else, then the search engine is wasting your time. 

Which brings us the broader point: when users search, they want to spend the least amount of time possible getting to the information they want. Relevance is a means to this end. In particular, clicks and typing costs users time. That time can come from page load, rendering, repeated use of the back button, and of course typing (and re-typing) search queries.

Some application search engines really nail the user experience. Let’s say we’re looking for the movie Koyaanits-however-you-spell-it. Go to the Internet Movie Database (IMDB) and start typing k-o-y-e — and there it is, as the second result. Notice that there is a ton of irrelevant stuff around it but it doesn’t matter. I see what I want very quickly.

Hopefully these two examples serve to illustrate the broader point: search engines should not focus on relevance as an end in itself, but rather on whatever helps users find the information they want as quickly as possible. That means offering contextual snippets, instant feedback, and of course snappy response times. Give users speed, and you will make them happy.

When I left Google last December, it was an open secret that Google was developing a social networking product. Now that Google has released Google+, I am at liberty to share my personal impressions.

Let’s start with the clear wins.

• Impressive launch. Google has certainly learned its lesson from the past launches of Wave and Buzz. Google+ is unambiguously opt-in — no one is going to complain about being ambushed. People have been begging for invites. But Google is wisely releasing invites quickly enough to build critical mass. I’d say that Google has at least picked up the Quora crowd of early adopters in Silicon Valley.

• Clean design. Design lead Andy Hertzfeld (of Macintosh fame) has nailed it, leading bloggers to comment that this looks too well designed to be a Google product. Comparing Google+ to Facebook now, I’m reminded at least a little of comparisons between Facebook and Myspace. Great move for Google here.

Now let’s talk about Google’s three big features here: Circles, Sparks, and Hangouts.

• Circles. Straight out of Paul Adams’s presentation of social networking (which he created before he left Google for Facebook), the idea is simple: a person doesn’t have a single group of friends, but rather several groups that tend are mostly disjoint. Through Circles, Google+ makes this soft partitioning of the social space a core design principle. You add people to one or more circles, follow the stream of activity from a circle, and share with circles. It’s great in theory. But in practice it creates friction, especially for people trained on Facebook. There’s a trade-off between simplicity and expressive power, and Google is placing a strong bet on how users will make this trade-off.  I’m inclined to agree with Yishan Wong that “the sorting of friends into buckets (friend lists) is something that only nerds do”. Given Google’s deep expertise in machine learning, I’m expecting Google to reduce this friction by give users intelligent suggestions. Full disclosure: my colleagues at LinkedIn built InMaps, which infers communities from your social network.

• Sparks. The tagline for Sparks is “For nerding out. Together.” It feels like a positioning designed by Googlers for Googlers– you can see promotional videos here and here. I haven’t seen much talk about Sparks, and what little commentary I’ve seen is less than gushing. I’ve experimented with it a bit from a consumption side, and I confess I’m underwhelmed. Perhaps it’s a chicken-and-egg problem — Sparks will only be useful if users populate their profiles with interests, but right now users have no incentive to do so. If Sparks is Google’s attempt to make Reader more social, there’s still a ways to go. Full disclosure: LinkedIn has its own approach to social news, LinkedIn Today, which seems to be doing something right.  

• Hangouts. In plain English, Hangouts are group video chat embedded in a social network. Which sounds a lot like what Facebook is rumored to be releasing this week through a partnership with Skype. Which in turn was just acquired by Microsoft. Will Apple join the party too by implementing group chat in FaceTime? Competitive dynamics aside, this is a very cool feature that hopefully won’t devolve into Chatroulette. Nothing to, um, disclose here.

But the $64B question is whether all this will matter. Can Google+ sustainably co-exist with Facebook? Will people use both services — and, if so, how will they allocate their attention between them? Or is the success of Google+ predicated on displacing Facebook? Or Twitter? Either of those would certainly qualify as a Big Hairy Audacious Goal.

Like Fred Wilson, I’m rooting for Google+ to succeed — but even Fred notes that he would not be able to get his family on Google+, as they are already happy with Facebook. It’s not clear to me what I can get *today* from Google+ that I can’t get from Facebook.

Granted, I’m not a heavy Facebook user, so I’m not the best person to ask this question. So readers, I ask you: why will or won’t you use Google+?

Don’t worry, it’s safe for work. Well, unless you work for Linden Lab.

Given that I work for the world’s largest professional network, I take work very personally. I’m also deeply involved in LinkedIn’s hiring process, which gives me opportunities to see how people make career decisions. I thought I’d share my own perspective here.

For me there are three things that matter to me about my work:

1. Do I love the work I do? Does work feel like play, stimulating me intellectually and emotionally? Am I excited about the people I work with? Is work a grind, or is it something I do for fun?
2. Is the work I do of value to my employer? Am I justifying my employer’s investment in me, or am I a freeloader lost in the inefficiency of corporate bureaucracy?
3. Is my work making the world a better place? Specifically, is the work I do making the world by more like the world I want to live in?

Not everyone may share my above values, and in any case not every job can address all of these values. But I am fortunate to have found one that does, and I’m loving it. To borrow a phrase, it just works.

If you haven’t seen this video by Dan Pink on what motivates people, I urge you to watch it. It’s a great reminder that there is more to motivation than economic incentives.

Finally, I hope that you are doing work that fulfills you. As I work to grow my great team at LinkedIn, my mission is not only to to bring great people to LinkedIn, but bring great work and fulfillment to great people. Whatever you do, be amazing.

Last weekend I had the extraordinary privilege to attend Foo Camp, an annual gathering of about 250 Friends Of O’Reilly (aka Foo). Tim O’Reilly, Sara Winge, and their colleagues have amazing friends, as you can see if you scan this unofficial list of attendees working on big data, open government, computer security, and more generally on the cutting edge of technology and culture (especially where the two overlap).

Foo Camp is an unconference, which merits some elaboration. No fees, no conference hotel (many attendees literally set up camp in the space O’Reilly provided), and no advance program aside from some preselected 5-minute Ignite presentations. Attendees proposed and organized sessions, merging and re-arranging them to optimize for participation. It was a bit chaotic (especially the mad rush after dinner to secure session slots), but very effective.

The minimalist format brought out the best in participants.

For example, I am passionate about (i.e., against) software patents, so I organized a session about them. I did a double-take when I realized that one of the participants was Pamela Samuelson, perhaps the world’s top expers on intellectual property law. I braced myself to be schooled — as I was. But she did it gently and constructively. Specifically, she pointed me to work that her colleagues Jason Schultz and Jennifer Urban were doing on a defensive patent strategy for open-source software (including a proposed license), as well as reminding me of the Berkeley Patent Survey supporting the argument that software entrepreneurs only file for patents because of real or perceived pressure from their investors. I also heard war stories from lawyers who have done pro bono work against patent trolls, reinforcing my own resolve and also reassuring me that the examples I’ve seen at close range are not isolated.

Another session asked whether we are too data driven in our work. What was notable is that this session included participants from some of the largest internet companies debating some of the must fundamental ways in which we work, e.g., do we actually learn from data or do we engage in assault by data to defend preconceived positions (cf. argumentative theory). Like all of the conference, the discussion was under “frieNDA”. so I’m being intentionally vague on the specifics. But it was refreshing to see candid admission that all of us know and have experienced the dangers of manipulating an audience with data, and that there are no algorithms to enforce common sense and good faith.

I won’t even try to enumerate the sessions and side conversations that excited me — topics included privacy, the future of publishing, a critical analysis of geek culture, and irrational user behavior. I missed the session on data-driven parenting, though others have pointed out to me that you can only learn so much if you don’t have twins and perform A/B tests. The best summary is intellectual diversity and overstimulation. If you’d like to get a general sense of the discussion, check out the #foocamp tweet stream. I also recommend Scott Berkun’s post on “What I learned at FOO Camp“.

As someone who organizes the occasional event, I’m intrigued by the unconference approach — especially now that I’ve experienced it first-hand. Moreover, I feel strongly that the academic conference model needs an upgrade. But I also know that open-ended, free-form discussion sessions are not a viable alternative — indeed, a big part of Foo Camp’s success was how it inspired participants to organize sessions — and to vote with their feet to attend the worthwhile ones. And of course part of that success came from inviting active, engaged participants rather than passive spectators.

Many of you also organize events, and I’m sure that all of you attend them. I’m curious to hear your thoughts about how to make them better, and happy to share more of what I learned at Foo Camp. After all, Foo is for (inspiring) thought.

As promised, here is a video of CMU professor Christos Faloutsos‘s recent tech talk at LinkedIn on “Mining Billion-Node Graphs“. Enjoy!

And check out our next week’s open tech talk by Sreenivas Gollapudi of Microsoft Research on “A Framework for Result Diversification in Search“.

ps. If you like these topics, then please talk to me about opportunities at LinkedIn! My group is hiring, as are many others.

The war for talent. It’s the latest metaphor for the challenge that tech companies face as excitement is building in Silicon Valley again. Well, not really — McKinsey coined the phrase in 1997 and used it as the title of a book published four years later.

But anyone who has been trying to hire great software engineers in recent months knows how hard it is to do so. Particularly for folks like me who are trying to hire data scientists — apparently there’s a national shortage. This is nothing new — as Joel Spolsky noted in a 2006 post, “the great software developers, indeed, the best people in every field, are quite simply never on the market.”

I’m not an expert (or ninja) on the subject of recruiting or employer branding in general, but I’ve seen enough of how companies go about hiring software engineers to know that we can do better. I’d like to share some of my thoughts and experiences, and I hope that you will reciprocate and share your thoughts in the comments. I’m especially interesting in hearing from folks who are at universities (aka hunting grounds) or who are involved in organizing academic conferences.

First, let’s talk about how we measure success. As Lord Kelvin famously said, “If you can’t measure it, you can’t improve it.” I’m not going to talk about how to handle active candidates — that’s a filtering problem which, in my opinion, is much more tractable. For example, see what Joel has to say about interviewing developers. Rather, I’m concerned with the challenge of discovering qualified passive candidates and converting them into active ones. Hence, I propose we make our metric the number of qualified applicants.

The baseline strategy is sourcing, i.e. have sourcers or hiring managers scour the world for qualified candidates (there’s an app for that), entice them with your best recruiting pitch, and then go hog wild on the folks who respond. The success of this strategy depends mainly on the rate at which you, your sourcers, or your hiring managers find qualified candidates — which in turn may split into the two subtasks of finding candidates and filtering them — and the conversion rate for the qualified candidates you find. Since the best candidates are often happy in their current positions, sourcing passive candidates requires a lot of work and a thick skin for rejection.

What are other ways to attract qualified passive candidates? Here are a few, with examples from my experience at LinkedIn:

• Hosting events. Last week at LinkedIn, we hosted CMU professor Christos Faloutsos, who delivered a fantastic talk on “Mining Billion Node Graphs” — a topic we thought interesting enough to justify opening up the talk to the general public. We had a few hundred guests, many of whom are precisely the kinds of folks we are trying to hire. Even more people watched the live stream online or will watch the video when we post it to YouTube (coming soon — stay tuned!). While this was not a recruiting event (we did not even announce that we are hiring), it was a great opportunity to associate LinkedIn with the hard computer science problems we solve on a daily basis.
• Sponsoring events. Sponsorship is tricky — if you’re not careful, you spend a lot of money for a glorified display ad. Sometimes sponsorship offers speaking slots as part of the package, but audiences are rightfully skeptical of speakers who have paid for their slots — especially at conferences that charge hefty fees for attendance. But sometimes sponsorship works. For example, LinkedIn’s was a sponsor of the O’Reilly Strata Conference, and the perks of sponsorship complemented our earned speaker slots, helping us bring enormous visibility to our data scientist team and its recent innovations like InMaps (we has a booth there to print attendees’ InMaps) and Skills (which launched during the conference). While Strata generated few direct leads, it left a lasting impression in the big data community, and I regularly hear candidates refer to it.
• Participating in events. As the Beatles tell us, money can’t buy you love. If you want to make an (positive) impression at a conference, you have to contribute people and ideas. This is especially true at academic conferences, where attendees quickly throw out the the extra weight in their tote bags and focus on the conference’s content and professional networking opportunities. It’s great if you are Microsoft with a team of close to a thousand researchers and can dominate a conference like SIGIR. But smaller companies can still make a strong impression on researchers — and especially on students who may be looking for internships or full-time positions — by taking an active role at conferences. The traditional approach is to submit papers to the main conference track — but other avenues include tutorials, workshops, and industry events. Such participation is often invited, but such invitations are in turn earned by cultivating relationships with researchers — especially the ones who find themselves on organizing committees.
• Contribute to open source projects. The Search, Network, and Analytics (SNA) team at LinkedIn contributes frequently to open-source projects and publicizes some of its work at http://sna-projects.com/. Open source projects are a great way to earn the respect of engineers who value source over PowerPoint. Especially when your employees include committers to key technologies like Hadoop. Moreover, open-source projects are social communities, so contributing to them offers opportunities for employees to interact with potential hires.
• Social media. By now, I’d like to think that marketers understand social media to simply be another set of marketing channels. But I think the territory is still pretty new for employers. Here is a simple suggestion: encourage (but do not try to force) employees to express themselves professionally online. Enforce the standard non-disclosure rules, of course, but don’t try to manage their voices. Authenticity speaks for itself — for example, look at what Adam Nash says about LinkedIn on his personal blog. Or my own posts here. Engineers don’t read press releases or  corporate blogs, but they do pay attention to their peers. And there’s nothing unique about blogs — the same principle applies to platforms like Twitter, Facebook, Quora, and of course LinkedIn. Not all employees enjoy being online extroverts, but those that do not only act as brand ambassadors, but also are likely to eventually strike up conversations with passive candidates about employment opportunities.

Finally, don’t forget measure the results of these efforts! Some activities generate leads directly, in which case you can make an apples-to-apples comparison of their results and costs with the baseline strategy of sourcing. It’s harder to measure the longer-term effect of efforts to raise visibility, but you can at least ask candidates if they are aware of those efforts — after all, efforts to raise visibility should be visible to candidates! You can also ask candidates if those efforts were a factor in their decision to apply. These measures aren’t perfect, but they are a lot better than nothing, especially when you’re trying to decide how best to invest limited resources.

Of course, even an optimal strategy can’t substitute for offering a combination of interesting work, competitive compensation, and a work hard / play hard culture. As with all marketing efforts, you need to start with a great product. But great products don’t sell themselves: you need to invest in a combination of outbound and inbound marketing to have a fighting chance in the war for talent. Good luck! And, in case you didn’t notice, we’re hiring!

If you Google [relevance theory], you’ll discover this Wikipedia entry about a theory proposed by Dan Sperber and Deirdre Wilson arguing that, in any given communication situation, the listener will stop processing as soon as he or she has found meaning that fits his or her expectation of relevance. The Wikipedia entry offers the following example of this principle:

Mary: Would you like to come for a run?

Bill: I’m resting today.

We understand from this example that Bill does not want to go for a run. But that is not what he said. He only said enough for Mary to add the context-mediated information: i.e. someone who is resting doesn’t usually go for a run. The implication is that Bill doesn’t want to go for a run today.

This theory may call to mind the Gricean Maxims — indeed, Sperber and Wilson borrow heavily from Grice’s work.

But I mainly bring up relevance theory to introduce Sperber to those unfamiliar with him. My friend (and Endeca co-founder) Pete Bell recently called to my intention an article by neuroscientist Jonah Lehrer entitled “The Reason We Reason“. The article reviews the “hot hand” fallacy and then proceeds to cite a new theory by Sperber and Hugo Mercier:

Reasoning is generally seen as a mean to improve knowledge and make better decisions. Much evidence, however, shows that reasoning often leads to epistemic distortions and poor decisions. This suggests rethinking the function of reasoning. Our hypothesis is that the function of reasoning is argumentative. It is to devise and evaluate arguments intended to persuade.

The full article by Mercier and Sperber runs over 17K works and is entitled “Why do humans reason? Arguments for an argumentative theory“.

As someone who has spent most of his professional life thinking about information retrieval in practical contexts, I automatically relate relevance theory to relevance in the context of information retrieval. Relevance has been a subject of intense debate in the information science community (Tefko Saracevic tells the story wonderfully). Indeed, a key reason that I created the HCIR workshop was the belief that information retrieval researchers and practitioners (i.e., search engine developers) were placing too much emphasis on an objective notion of topical relevance, and not enough focus on the user.

Mercier and Sperber’s theory offers an interesting challenge to information retrieval researchers: perhaps a user’s information need is less about arriving at the truth and more about finding confirmatory evidence to support a preconceived conclusion. If so, should we adjust our notions of relevance accordingly? Also, if we evaluate or inform search quality based on observed user behavior (such as click-through behavior), then are we already inadvertently conflating topical relevance with users’ confirmatory bias?

Many people have noted that personalization gives us the truth we want: recent examples include Robin Sloan and Matt Thompson’s EPIC 2014 and Eli Pariser’s The Filter Bubble. Despite the consensus that over-fitting information access to our personal tastes is a bad thing (perhaps even dystopian), technology seems to relentlessly push us in this direction. Moreover, some degree of personalization is clearly useful — such as prioritizing information that relates to our personal and professional interests.

Nonetheless, anyone working in the area of information seeking systems should be concerned with the question of the user’s goal in using that system. Many of us take for granted that the user’s main goal is truth seeking, and we design our systems accordingly. What can or should we do differently if the user’s main goal is not informative but persuasive? Is the user looking for an answer…or an argument?

What a day! I’ve been excited about LinkedIn from the moment I joined — and for several years before that — but today has been a unique experience. I hope our celebration extends beyond LinkedIn’s employees and investors — this is a great day for Silicon Valley, for the data scientists who are building its most valuable companies, and for the users who are benefiting from it all. I am proud and deeply grateful to be a part of this extraordinary adventure. My thanks to my hundreds of incredible colleagues and to the 100M users who have made it possible.

ps. Yes, we are still hiring, so please contact me if you’re the kind of person who loves turning data into gold. And if you are local, check out Christos Faloutsos’s upcoming tech talk on Mining Billion Node Graphs, which will take place at LinkedIn on June 2 and is open to the public.

A couple of weeks ago, I participated in a summit that Greylock Partners organized for its portfolio companies at LinkedIn to discuss the power of data. Invited participants represented some of the most interesting “big data” companies in Silicon Valley, including Google, Facebook, Pandora, Cloudera, and Zynga. Discussion took place under the Chatham House Rule, so I’m not at liberty to share much detail. But I can say that there were energetic conversations about metrics, tools, and (of course) hiring.

One of the participants was Google researcher Alon Halevy, who generously shared his presentation on Fusion Tables with me with permission to re-share it here.

Fusion Tables allow the general public to upload, visualize, and share structured data. They are particularly useful for journalists who want to distill compelling stories from data — indeed, The Guardian‘s Simon Rogers has used Fusion Tables to visualize and interpret everything from nuclear power plant accidents to Wikileaks.

After his presentation, I asked Alon for his thoughts on why haven’t we seen an encyclopedic structured data repository comparable in scope, scale to Wikipedia? Alon offered that structured data is brittle — its value tends depend more on context than the unstructured content that populates Wikipedia. I agree in part — for example, consider this map of Brooklyn bus stops that were slated for elimination last summer. Such data is useful in a narrow context, but hardly encyclopedic.

But what about Freebase and DBpedia? Freebase is an open repository of structured data associated with about 20 million topics. DBpedia describes itself as “a community effort to extract structured information from Wikipedia and to make this information available on the Web.” While these tools have seen some use by developers (especially in the semantic web community), they have not achieved mainstream adoption. Perhaps data marketplaces like Factual and Infochimps will be successful as for-profite businesses, but the question remains why we don’t have a Wikipedia-scale success story for public structured data.

I think the problem is easiest to frame in information retrieval terms. Wikipedia is all about precision, but not so much about recall. Let me elaborate.

Wikipedia represents a collective attempt to achieve precision at the level of individual entries. Contributor / editors correct mistakes and argue over the details of content and tone. But coverage is a much lower priority. When in doubt, the Wikipedia collective assumes that information is not notable enough to justify inclusion. Thus Wikipedia errs on the side of precision rather than recall when it comes to meeting the information needs of its users.

This arrangement works well for a typical web user who seeks out information by using Google web search as an interface to discover Wikipedia articles. But structured data is about sets, not just individuals. It does me no good to see aggregate statistics about a set of entities if the set is erratically populated (e.g., Wikipedia’s list of companies established in 1999 or Freebase’s list of those founded after 2000).

In the June 2009 SIGIR Forum, University of Melbourne researchers Justin Zobel, Alistair Moffat, and Laurence Park argued “against recall“, concluding that they could find “no justification for implicit or explicit use of recall as a measure of search satisfaction.” I posted a rebuttal entitled “In Defense of Recall“, arguing that recall is much more useful as a measure for set retrieval than for ranked retrieval. Revisiting this argument two year later, I can see that it holds even more strongly if we are interested in structured data where we want to reason about aggregate properties of sets.

Back when we both worked at Endeca, my colleague Rob Gonzalez described structured data repositories to be as a public good that no one is ever willing to pay for. I’m an optimist by nature, but in this case I fear he has a point. It takes a lot of work to build something useful, and no one seems to have addressed the challenge of incenting people to contribute this work for either economic or altruistic motives.

Or perhaps we’ll just have to wait for the holy grail of information extraction algorithms to structure the world’s information for us? Ironically, that’s not even included on Wikipedia’s list of AI-complete problems.

As regular readers know, I’ve been co-organizing annual workshops on Human-Computer Interaction and Information Retrieval since creating the first HCIR workshop in 2007. These have been a huge success, not only bridging the gap between IR and HCI, but also bringing together researchers and practitioners to address concerns shared by both communities. Past keynote speakers have included such information science luminaries as Susan Dumais, Ben Shneiderman, and Dan Russell.

Every workshop has improved on the previous year’s, and HCIR 2011, which will take place on Thursday, October 20, will be no exception.

Our venue will be Google’s headquarters in Mountain View, California. We could hardly imagine a more appropriate venue: Google has done more than any another company to contribute to everyday information access. Google has been extremely generous as a host and sponsor (other sponsors include Endeca and Microsoft Research), and its location in the heart of Silicon Valley is ideal for attracting researchers and practitioners building the future of HCIR.

Our keynote speaker will be Gary Marchionini, Dean of the School of Information and Library Science at the University of North Carolina at Chapel Hill. Gary coined the phrase ”human–computer information retrieval” in a lecture entitled “Toward Human-Computer Information retrieval“, in which he asserted that ”HCIR aims to empower people to explore large-scale information bases but demands that people also take responsibility for this control by expending cognitive and physical energy.” We are honored to have Gary deliver this year’s keynote.

But of course the main attraction is the contribution of participants. This year we invite three types of papers: position papers, research papers and challenge reports. Possible topics for discussion and presentation at the workshop include, but are not limited to:

• Novel interaction techniques for information retrieval.
• Modeling and evaluation of interactive information retrieval.
• Exploratory search and information discovery.
• Information visualization and visual analytics.
• Applications of HCI techniques to information retrieval needs in specific domains.
• Ethnography and user studies relevant to information retrieval and access.
• Scale and efficiency considerations for interactive information retrieval systems.
• Relevance feedback and active learning approaches for information retrieval.

Demonstrations of systems and prototypes are particularly welcome.

Building on the success of the last year’s HCIR Challenge to address historical exploration of a news archive, this year’s HCIR Challenge will focus on the problem of information availability. The corpus for the Challenge will be the CiteSeer digital library of scientific literature.

For more information about the workshop, including how to submit papers or participate in the challenge, please visit the HCIR 2011 website.

Here are the key dates for submitting position and research papers:

• Submission deadline (position and research papers): July 31
• Notification of acceptance decision: September 8
• Presentations and poster session at workshop: October 20

Key dates for Challenge participants:

• Request access to corpus (contact me) deadline: June 19
• Freeze system and submit brief description: September 25
• Submit videos or screenshots demonstrating systems on example tasks: October 9
• Live demonstrations at workshop: October 20

I’m looking forward to this year’s submissions, and to a great workshop in October. I hope to see many of you there!

As I posted a few months ago, I’m organizing the Industry Event at CIKM 2011 with Tony Russell-Rose. We have a great set of keynotes lined up:

• Stephen Robertson (Microsoft Research)
• John Giannandrea (Google)
• Jeff Hammerbacher (Cloudera)
• Peter Jackson (Thomson Reuters)

We’re also looking for submissions from industry researchers and practitioners. The submission deadline is June 21.

Here is a copy of the call for papers:

This year’s CIKM conference will include an Industry Event, which will be held during the regular conference program in parallel with the technical tracks.

The Industry Event’s objectives are twofold. The first objective is to present the state-of-the-art in information retrieval, knowledge management, databases, and data mining, delivered as keynote talks by influential technical leaders who work in industry. The second objective is to present interesting, novel and innovative industry developments in these areas.

Industry authors are invited to prepare proposals for presenting interesting, novel and innovative ideas, and submit these to industry@cikm2011.org by June 21st 2011. The proposals should contain (with respective lengths):

• Short company portrait (125 words)
• Short CV of the presenter (125 words)
• Title and abstract of the presentation (250 words)
• Reasons why the presentation should be interesting to the CIKM audience

When submitting a proposal, please bear in mind the following:

• Ensure the presentation is relevant to the CIKM audience (the Call for Papers gives a good idea of the conference scope).
• Try to highlight interesting R&D challenges in the work you present. Please do not present a sales pitch.
• All slides will be made public (no confidential information on the slides; you will be expected to ensure your slides are approved by your company before being presented).
• Presenters may opt to have their presentation videoed and made public, and if so, the presenter will be asked to sign a release form.

We look forward to receiving your submissions, and welcoming you to the CIKM 2011 Conference and Industry Event.

Important dates:
21 June 2011: Industry Event paper proposals due
19 July 2011: Notifications sent
27 October 2011: Industry Event
24-28 October 2011: CIKM conference

Pankaj Mehra and I are guest editors for an upcoming special issue of IEEE Internet Computing with the topic “Beyond Search: Context-Aware Computing“.

Here is a copy of the call for papers:

Context is the unstated actor in human communications, actions, and situations. It makes our communication efficient, our commands actionable, and our situations understandable to the people, organizations, and devices that provide us with content or services. The increased embedding of technology into our personal and social environments drives a need for context-aware computing.

Context-aware computing offers mobile Internet users an experience that goes beyond user-initiated search and location-­based services. Context awareness sharpens relevance when responding to user-initiated actions (such as product search and support calls). It also enables proactive communications through analysis of a user’s behavior and environment, thereby forming the basis for key business imperatives targeting customer-engagement systems. Even greater opportunity arises from context use in systems that can make sense of and engage in customer dialogs and forums.

This special issue seeks original articles that support and illustrate context use in creating enhanced user experiences. Sample topics include

• proactive, contextualized delivery of information, alerts, and advertisements;
• context-mediated Web service orchestration, yielding actionable interpretation of spoken high-level commands;
• system architecture, economics, and ecosystems for comprehensive capture, representation, communication, gathering, and brokering the larger user context;
• systems of engagement that treat discourse as text plus context and process textual communication as an event in which linguistic, cognitive, and social actions converge; and
• reasoning and knowledge representation mechanisms that use context in selecting the body of knowledge to use, the level of detail to model, and the point of view with which to communicate and interpret text and data.

All submissions must be original manuscripts of fewer than 5,000 words, focused on Internet technologies and implementations. All manuscripts are subject to peer review on both technical merit and relevance to IC’s international readership—primarily system and software design engineers. We do not accept white papers, and we discourage strictly theoretical or mathematical papers. To submit a manuscript, please log on to ScholarOne (https://mc.manuscriptcentral.com:443/ic-cs) to create or access an account, which you can use to log on to IC’s Author Center and upload your submission.

I hope some of you will submit articles in time for the June 15 deadline, and Pankaj and I look forward to reviewing them.

I thought I’d prepare for the discussion by sharing my thoughts here. Perhaps some of you will even be kind enough to add your own ideas, which I promise to share with the reading group.

I encourage you to read the paper, but here’s a summary of its results:

• A user’s influence on Twitter is the extent to which that user can cause diffusion a posted URL, as measured by reposts propagated through follower edges in Twitter’s directed social graph.
• The best predictors of future total influence are follower count and past local influence, where local influence refers to the average number of reposts by that user’s immediate followers, and total influence refers to average total cascade size.
• The content features of individual posts do not have identifiable predictive value.
• Barring a high per-influencer acquisition cost, the most cost-effective strategy for buying influence is to target users of average influence.

Let’s dive in a bit deeper.

The definitions of influence and influencers are, by the authors’ own admission, narrow and arbitrary. There are many ways one could define influence, even within the context of Twitter use. But I agree with the authors that these definitions have enough verisimilitude to be useful, and their simplicity facilitates quantitative analysis.

It’s hardly surprising that past influence is a strong predictor of future influence. But it might seem counterintuitive that, for predicting future total influence, past local influence is more informative than past total influence. The authors suggest the explanation that most non-trivial cascades are of depth 1 — i.e., total influence is mostly local influence. But at most that would make the two features equally informative, and total influence should still be a mildly better predictor.

I suspect that another factor is in play — namely, that the difference between local influence and total influence reflects the unpredictable and rare virality of the content (e.g., a random Facebook Question generated 4M votes). If this hypothesis is correct, then past local influence factors out this unpredictable factor and is thus a better predictor of both future local influence and future total influence.

I’m a bit surprised that follower count supplies additional informative value beyond the past local influence; after all, local influence should already reflect the extent to which the followers are being influenced. It’s possible that past influence lags the follower count, since it does not sufficiently weigh the potential contributions of more recent followers. But another possibility is one analogous to the predictive value of past local vs. global influence: past local influence may include an unpredictable content factor which follower count factors out.

Of course, I can’t help suggesting that TunkRank might be a more useful indicator than follower count. Unfortunately the authors don’t seem to be aware of the TunkRank work — or perhaps they preferred to restrict their attention to basic features.

I’m not surprised by the inability to exploit content features to predict influence. If it were easy to generate viral content, everyone would do it. Granted, a deeper analysis might squeeze out a few features (like those suggested in the Buddy Media report), but I don’t think there are any silver bullets here.

Finally, the authors consider the question of designing a cost-effective strategy to buy influence. The authors assume that the cost of buying influence can be modeled in terms of two parameters: a per-influencer acquisition cost (which is the same for each influencer) and a per-follower cost for each influencer. They conclude that, until the acquisition cost is extremely high (i.e., over 10,000 times the per-follower cost), the most cost-efficient influencers are those of average influence. In other words, there’s no reason to target the small number of highly influential users.

The authors may be arriving at the right conclusion (Watts’s earlier work with Peter Dodds, which the paper cites, questions the “influentials” hypothesis), but I’m not convinced by their economic model of an influence market. It may be the case that professional influencers are trying to peddle their followers’ attention on a per-follower basis — there are sites that offer this model.

But why should anyone believe that an influencer’s value is proportional to his or her number of followers? The authors’ own work suggests that past local influence is a more valuable predictor than follower count, and again they might want to look at TunkRank.

Regardless, I’m not surprised that a fixed per-follower cost makes users with high follower counts less cost-effective, as I subscribe to its corollary: as a user’s follower count goes up, the per-follower value diminishes. I haven’t done the analysis, but I believe that the ratio of a user’s TunkRank to the user’s follower count tends to go down as a user’s follower count goes up. A more interesting research (and practical) question would be to establish a correctly calibrated model of influencer value and then explore portfolio strategies.

In any case, it’s an interesting paper, and I look forward to discussing it with my colleagues next week. Of course, I’m happy to discuss it here in the meantime. If you’re in my reading group, feel free to chime in. And you’re not in you’re not in my reading group, consider joining. We do have openings.

I like Google…

I’ve been a regular Google user since the day I first discovered its existence in 1999. Indeed, I’ve consistently found Google to be the most useful service on the web. That’s not love, but it’s a very strong +1.

Moreover, I’d say that my preference for Google is an informed one. I’ve given all of the major search engines a fair chance, and even tried a fair number of obscure ones. They all have their strengths, but none have delivered enough utility to me to justify the cognitive load of using more than one search engine for the open web.

…but I don’t need Google.

Nonetheless, I know that, if Google disappeared tomorrow or became inconvenient to access, I’d be content with one of its competitors. I have no particular investment in Google beyond brand loyalty.

Actually, that’s not entirely true. I could easily walk away from Google search, but I’d be apoplectic if I suddenly lost access to my Gmail account — much as if I lost access to my LinkedIn or Twitter accounts. Indeed, Gmail is the only way in which Google has me locked in, but I don’t see my Gmail account as entangled with my access to Google’s other services.

Perhaps that not a bug but a feature: after all, Google trumpets the virtues of “open” and the portability of user data (including Gmail) through the Data Liberation Front. Nonetheless, it’s no secret that Google has a major case of Facebook envy. And if rumors hold, Google is now making the success of its social strategy a major component in all employee compensation.

Social is Give to Get.

Google critics often assert that Google doesn’t get social. But I think the problem isn’t so much with what Google gets as what it gives. When it comes to social, you have to give to get. That is, to get data and engagement, you have to provide social utility.

To start off, Google would love to know who you are. That’s why it developed Google Profiles in 2007. People are more than willing to provide data about who they are, as proven by the hundreds of millions of people who create profiles on Facebook and LinkedIn. Perhaps Google was a little bit late to the game. More likely, people didn’t see enough utility in creating Google profiles. Facebook, on the other hand, helps people be found by their friends and family in a context designed for social interaction. LinkedIn offers people the opportunity to be found by people who can help you professionally: colleagues, classmates, potential employers, etc. Google didn’t give people much reason to invest effort — in fact it seems to treat Profiles as a dumping ground populated by Google’s other products, rather than valuable piece of online real estate embedded in a living social context. Not surprisingly, users invest their efforts elsewhere.

Google would also love to know where you are and where you’ve been — that’s why Google created Latitude in 2009. Moreover, Google developed this pioneering location-based service as a complement to Google Maps, perhaps the best product Google has produced outside of search. Given it’s dominance in mapping services, directions, and local search, Google should be the leader of all things local. And yet, while Latitude has flopped, Foursquare — which launched in the same year as a tiny startup after Google acquired and shut down its previous incarnation– succeeded in defining location-based services as a category. Before Foursquare, the idea of a service tracking your location was one that most of us associated with Lo-Jack and Big Brother — if not with modern totalitarian regimes. Yet, by making a game out of “checking in” to venues, Foursquare inspired its users to willingly — and eagerly! — share and publish their whereabouts. It’s unclear whether this model will create sustained interest (cf. Mark Watkins’s analysis at ReadWriteWeb), but Foursquare’s success thus far is predicate on its offers social utility in exchange for data and attention.

Of course, Google also wants to know what you like. That’s why Google developed SearchWiki (RIP), Hotpot (now merged into Places), and most recently +1. As Amazon, Facebook, Netflix, and Yelp have demonstrated, people aren’t shy about sharing their opinions publicly, given the right social context and utility. Unfortunately, Google seems to struggle with that last part. Google embedded SearchWiki in the non-social context of search — and has launched +1 the same way. It’s not at all clear what users would gain by going out of their flow to annotate search results. Hotpot may simply be a case of too little, too late — people are already trained to go to Yelp and Facebook Fan pages for subjective information about service businesses. Overall, Google has not given users a reason to believe there is significant return on their investment in sharing opinions.

Collecting Data Doesn’t Count.

Of course Google is able to collect a significant amount of data about users’ identities through their search history, cookies, browser toolbars, and purchase history (if they use Google Checkout). Indeed, it is Google inference of user intent in search queries that has allowed Google to become the poster child of online advertising.

But collecting data is not the same as having the user volunteer it. Most users have a transactional relationship with Google, tolerating data collection and advertising in exchange for a free service. Google wants more — it wants users to invest in identities associated with their Google accounts. But Google doesn’t seem to undertand that users don’t make these investments unless their receive some social or professional utility in return.

If it’s true that Larry Page is making “social” Google’s top OKR, then I hope for the sake of my former colleagues that Google has learned from its past experiments.

Note: This post was written by Scott Nicholson, a Senior Data Scientist at LinkedIn. Scott is data and modeling geek with a passion for startups, product and user experience. His work at LinkedIn focuses on analyzing and improving user engagement and monetization.

I’m happy to report back on my experience at the Data 2.0 conference, an event organized by midVentures and targeted at entrepreneurs building products to leverage the dramatic increase in publicly and privately collected data. The conference has four main themes: what data is available, how to obtain data, how to store and access data, and how to create value from data products. For data nerds or hackers, the conference offered a delightful stream of  “you know what would be cool…” ideas.

The morning started off on a strong foot with a talk by Vivek Wadhwa on how data is going to define the next generation of successful startups in a new information age. He observed the increasing online access to data that has previously been restricted to offline access (or no access at all). He also emphasized the importance of  new sources of data, such as medical records and genome data. We need to think of social use of data beyond Twitter, Facebook and LinkedIn: for example, genome data will allow us to connect to each other in ways that helps us better understand our similarities and differences. Meanwhile, some existing data sources will become increasingly open and available to all. Wadhwa stressed the importance of leveraging the open sources of federal, state and local government data to come up with solutions to the existing closed and clunky legacy systems that governments used to generate data reports (a pity that data.gov and related programs may be defunded — DT).

The morning keynote segued nicely into the panel on open data sources. Jay Nath, Director of CRM for the city of San Francisco, noted that, while many applications are using government data and APIs, they mostly address consumer convenience (e.g., public transit apps) rather than government efficiency.  Panelists agreed that government employees have few incentives to take risks by using new technology: legacy systems might be expensive, inflexible and inefficient, but they do perform their limited function. Alluding to Eric Ries’s idea of a “lean startup“, Nath suggested the concept of a “lean government” that lowered costs, sped up its operations, and avoided procurement processes by using open source technology — all in the context of providing services to its citizens.

The inspiring mid-day keynote by former Amazon Chief Scientist Andreas Weigend took a different perspective from the morning sessions: he focused on the how data sharing can provide tangible value to end-users, even resulting in significant behavior change. He cited products like tweeting weight scales, FitBit, and Nike + that allow people to share data about their fitness efforts, thus leading to social reinforcement for positive behaviors. I personally see this area as a great example of where data scientists and engineers can create enormous economic value and increase people’s welfare

The day also featured a various product launches and presentations. Here are a few that caught my attention:

• Micello: Google maps for indoors. They won the startup competition that was held in conjunction with the conference.
• Tropo: API for voice calls and SMS
• DataStax Brisk: Technology unifying Hadoop, Hive & Cassandra. A new Hadoop distribution powered by Cassandra.
• Neer: always-on location awareness app from Qualcomm. Privately share location with groups and families.
• Heritage Health Prize: $3MM prize for predictive modeling around who will require hospitalization (a follow-up on their announcing the prize at Strata)

Overall, it was great to see hundreds of people exploring innovations and opportunities to use data to improve business, technology and society.

My ideas are somewhat predictable, in that they all address user-centric information-seeking problems. Working for over a decade in this space has focused my intellectual curiosity somewhat — and of course I work on a number of these problems at LinkedIn. But there are many information-seeking problems that are outside of my present or foreseeable scope.

Here are two ideas that I’m hoping someone will execute on so I don’t have to:

1. Shopping: Help Me Figure Out What I Want

We’ve come a long way to improve the shopping experience, at least for utilitarian shoppers like yours truly. If I know exactly what I want, I usually find it by using Google as a gateway to Amazon, taking a bit more time if I’m feeling price-sensitive. I’d happily install a browser extension that could automatically detect product search queries and take them to my preferred shopping sites, bypassing the search results page, but that’s a minor detail of convenience (though probably not such a minor detail for the search engine companies). In any case, known-item search for online shopping is hardly inspiring as an open problem.

Exploratory search is another story entirely. For all the work that’s been done on faceted search, it is used almost exclusively to help people narrow search results. Progressive narrowing is great if you have a pre-established information need, but it is not the best interface if you’re hoping to evolve your information need through exploration. Instead of just “help me find what I’m looking for”, I’d also like to see more “help me figure out what I want”. I’d like to see an innovator applying faceted search to broaden queries, not just to narrow them, as well as going beyond collaborative filtering and “related items” to create a compelling browsing experience based on semantic and social navigation.

2. Organizing the World’s Information: Beyond Wikipedia and Navigational Queries

If shopping online often reduces to using Google to find product pages on Amazon, then informational queries similarly reduce to using Google to find Wikipedia entries. Nothing against Wikipedia — I think it is one of the most extraordinary achievements of our generation — but I think of the web as a library and Wikipedia as its encyclopedia section. Google’s mission statement notwithstanding, web search engines do a poor job of organizing the rest of the world’s information, instead choosing to optimize for known-item search.

There are countless opportunities for improvement here. Imagine if there were an interface for books, scholarly articles, patents, music, or videos that supported browsing and exploration of their content and meta-data. We’ve seen the beginnings of such an approach for individual libraries (e.g., the Triangle Research Libraries Network), but there is so much more to do in this space. Perhaps it’s a space that is hard to monetize, but even then I’d expect philanthropists to take an interest in making the world’s knowledge and creative artifacts more accessible.

If you are pursuing either of these areas, I’d love to hear about it. I’m sure readers here would too. I’m also curious to learn more about innovation in the travel and personals spaces, as those are both areas that could benefit from supporting exploratory search. And if you have work in progress, please present it at the HCIR workshop!

This afternoon, I met with a couple of Stanford seniors to advise them on a startup they’ve been developing and targeting towards mid-sized online retailers. I’d expected to spend most of the time talking about their technology and customer development strategy — and we did indeed talk about these things. But we spent most of the time brainstorming whom I knew that could best help them achieve the key milestone of landing a first customer.

Not surprisingly, my first step was to open up my laptop and head straight to LinkedIn (I’m not only a data scientist — I’m also a member!) to see who in my network might be most helpful to them at this critical stage. The students were openly impressed: despite being sharp, energetic, and remarkably business-savvy for a couple of guys not old enough to legally buy beer, they had never seen someone use LinkedIn the way I was doing in front of them — not for hiring or recruiting, but as an exploratory search tool to find useful professional connections.

I started with a search for online retail, then restricted to directors and VPs, narrowing down further to first-degree and second-degree connections. I vetted second-degree connections by looking at my paths to them, determining who would be likely to be most helpful either because they owed me a favor or because they might have their own interest in the startup’s success.

We then browsed through the list of top online retailers, identifying plausible companies for them to target and then looking for my first-degree and second-degree connections not only at those companies but also at other companies in the same space. We spent over an hour fluidly going back and forth between talking and exploring on LinkedIn. In the course of this exploration, we not only produced a list of people to contact, but also arrived at a better understanding of the business strategy.

I’m always happy to help young entrepreneurs who represent the future of our economy, and even happier to do so using the tools my colleagues and I are constantly working to improve. But I’m surprised and a bit disheartened that the methods I used are not common knowledge, especially among people who stand to gain the most benefit from them. Perhaps, as someone who has been using LinkedIn since 2004, I take for granted that people know how to take advantage of it for professional networking. I hope that the company’s increasing visibility will make more people aware that LinkedIn is not *just* the best things that has ever happened to recruiting.

Also, as an HCIR advocate, I’d like to see these kinds of information-seeking tasks receive more attention from researchers and practitioners. I’ve been saying for a while that these and similar tasks that are neglected by the information retrieval community and not adequately addressed by Google . For example, while there has been significant research effort in the area of expert finding, I’d like to see more efforts to improving the interactive process of finding experts and expertise. And not just from LinkedIn!

If you are doing work in this space, I hope you’ll participate in the upcoming HCIR workshop and show off your stuff.

In the meantime, I hope you make the most of LinkedIn, for fun and for profit. As a mentor of mine told me in my first job, it’s “network or not work”.

My feelings about software patents are a matter of public record (e.g., this open letter to the USPTO). As things stand today, software patents act as an innovation tax rather than as a catalyst for innovation. It may be possible to resolve the problems of software patents through aggressive reform, but it would be better to abolish software patents than to maintain the status quo.

My personal feelings notwithstanding, I acknowledge the reality that today’s software companies need to have defensive patent strategies. In a previous job, one of my key accomplishments was to hire a director of intellectual property. It was a difficult hire, but it happened just in time to defend against a particularly noxious patent troll. I am not at liberty to spell out the details, but I can say that we responded with a long, expensive fight that effectively quashed the patent and the lawsuit.

Beware Of Trolls

Patent trolls, known less pejoratively as non-practicing entities (NPE) because they do not actually sell products or services that implement the systems or methods in the patents they own, take advantage of asymmetric risk. On one hand, an NPE does not need much money to bankroll (or at least initiate) a patent infringement suit — in fact, there are law firms who will take such cases on contingency. On the other hand, the company being sued faces potentially ruinous costs. Moreover, even if a company feels certain that a lawsuit against it is baseless, the company cannot count on the imperfect and inefficient legal system to reach a fair outcome. As a result, the company has to choose between spending heavily in its own defense or settling with the NPE. Most companies opt for the less risky route and negotiate settlements, providing funds that the NPEs use to sue more companies.

Some people have a name for this style of asymmetric warfare — namely, terrorism. I suppose that the word terrorist is loaded enough without increasing its breadth to include patent trolls — not to mention that trolls have their defenders. But the metaphor is a useful one. A terrorist attack inflicts an amount of damage that is much greater than the absolute cost to the terrorist, e.g., a suicide bomber who inflicts mass murder. Moreover, the threat of terrorism puts the object of that threat in the position between settling (aka negotiating with terrorists) or spending heavily on counter-terrorism efforts. As Peter Neumann notes in a Foreign Affairs article:

The argument against negotiating with terrorists is simple: Democracies must never give in to violence, and terrorists must never be rewarded for using it. Negotiations give legitimacy to terrorists and their methods…

Yet in practice, democratic governments often negotiate with terrorists.

There have been various attempts to address the threat of patent trolls.

Google litigation director Catherine Lacavera has gone on record saying that Google intends to fight rather than settle patent infringement lawsuits in order to deter patent trolls. We’ll see if Google can sustain this “we don’t negotiate with terrorists” approach; I admire the resolve, but like Neumann I’m skeptical.

Article One Partners has built a business around crowd-sourcing patent invalidation. Clients pay for research to invalidate patents, and Article One offers bounties to anyone who contributes valuable evidence. In theory, companies can request validity analysis of their own patents to test them for robustness, but I assume that the primary application of this service is the invalidation patents that a company sees as threats.

Rational Patent (RPX) has created a defensive patent pool. purchasing a large portfolio of patents and then licensing them to its member companies. Some have questioned whether this approach is “patent extortion by another name“, and indeed paying RPX for a blanket license does feel a bit like preemptively settling in bulk. But I’d be more concerned that the “over 1,500 US and international patent assets” that RPX claims to have acquired are a drop in the bucket compared to the vast number of patents that the USPTO has granted, many of dubious merit.

Meanwhile, patent trolldom is serious business. Former Microsoft CTO Nathan Myhrvold created Intellectual Ventures to “invest both expertise and capital in the development and monetization of inventions and patent portfolios.” The company has only filed one lawsuit so far, but Mike Masnick claims that it has used over a thousand shell companies to conduct stealth lawsuits.

Unfortunately, the proliferation of lawsuits by software patent trolls suggests that the economic incentives encourage such suits. If every company could sustain a “Never give up, never surrender!” approach, patent trolls would eventually go away, but it is unlikely that companies would be willing to assume the short-term risks that such an approach entails.

Moreover, this approach only works if everyone participates, requiring that every company forgo the competitive advantage it could enjoy from being the only company among its competitors to appease the trolls. This is a classic tragedy of the commons. I’m hopeful that we’ll eventually implement sensible patent reform in the United States, but I expect it will take a long time to overcome the entrenched interests that support the status quo.

It’s Not Just The Trolls

But NPEs are not the only cause for concern. Many established companies, including some technology leaders, are not averse to using patent lawsuits as part of their business strategy. The mobile device and software space is a particularly popular arena for patent litigation, the most notable being Oracle’s lawsuit against Google claiming that Android infringes on patents related to Java. The stakes are extraordinary, dwarfing even the $612.5M that RIM paid NTP in order to avoid a complete shutdown of the BlackBerry service (ironically, at least some of the patents involved have since been rejected by the patent office after re-examination).

Patent lawsuits can also be a way for larger companies to bully smaller ones. For example, a couple of entrepreneurs at visual search startup Modista were forced to shut down their company because of a lawsuit by Like.com, a more established player in the space which was ultimately acquired by Google. Note: although I was Google at the time, I have no inside knowledge of the acquisition, nor whether there is any truth to the speculation that Google acquired the company for its patents.

Defensive Patenting

Moral considerations aside, the above stories make it clear that defensive patent strategy isn’t just about NPEs. In fact, many software companies take an approach to defensive patenting is to assemble a trove of patents that are useful for countersuits and thus serve as a deterrent. Back to military metaphors, it’s similar to countries developing nuclear weapons (a popular metaphor for patents in general) in accordance with the doctrine of mutual assured destruction.

Companies that follow a defensive patent strategy typically implement a process for capturing intellectual property. Scientists and engineers file invention disclosures, a committee reviews these for patentability, and a law firm translates the invention disclosures into patent filings. The filings then go through the meat grinder of patent prosecution and eventually are extruded as patents.

It all sounds great in theory — indeed, I have seen executives who mostly worry about educating scientists and engineers about patents and providing the right incentives to encourage them to write and submit invention disclosures. Indeed, it can be difficult to integrate intellectual property capture into the process and culture of a software company. But I think there are two much bigger issues.

First, it takes several years to obtain a patent. Indeed, the USPTO dashboard shows that it takes two years just to get an initial response from the patent office. Thus a defensive patenting strategy requires significant advance planning: any patents filed today are unlikely to be useful deterrents until at least 2014. Given the rapid pace of the software industry, this delay is very significant. Moreover, startups are especially vulnerable in their first few years.

Second, intellectual property capture processes are inherently optimized for offensive (i.e., don’t copy my invention or I’ll sue you) rather than defensive (i.e., don’t sue me) patent strategy. Consider Google’s defensive position with respect to Oracle in the aforementioned lawsuit. Google has a relatively small patent portfolio, but it has obtained patents for some of its major innovations, such as MapReduce. Let’s put aside questions about the validity of the MapReduce patent — especially since patents enjoy the presumption of validity. The bigger question is to whom such a patent serves as a deterrent against patent lawsuits. It may very well deter Hadoop users, which include Google arch-rival Facebook. But, as far as I know, Oracle is not vulnerable on this front. FOSS Patents blogger Florian Mueller did an analysis and concluded that Google’s patents are not an effective deterrent. Indeed, the fact that Google has not counter-sued Oracle using its own patents is at least consistent with this analysis.

What if Google were to invest in obtaining (i.e., purchasing) a collection of broad patents that had to do with relational databases? Such patents could have nothing to do with Google’s areas of innovation and nonetheless serve as an effective deterrent against lawsuits from relational database companies like Oracle. Even if the patents were not robust, they would still have some value as deterrents because of their presumption of validity and the aforementioned inefficiency of the legal system.

In general, the most valuable defensive patents are those that you believe your competitors (or anyone else who might have an interest in suing you) are already infringing. Even if those patents would be unlikely to survive re-examination, the re-examination process is long and expensive, and even the most outrageous of patents enjoys the presumption of validity.

Everybody Into The Pool

A patent pool is a consortium of at companies that agree to cross-license each other’s patents — a sort of mutual non-aggression pact. But perhaps companies that only believe in the defensive use of patents should take a more aggressive approach to patent pooling. Following the example of NATO, they could create an alliance in which they agree to mutual defense in response to an attack by any external party. I don’t know if such an approach would be viewed as anti-competitive, but it does strike me as a cost-effective alternative to the current approach for defensive patenting.

And, as with most ideas, this one is hardly original. In 1993, Autodesk founder John Walker published “PATO: Collective Security In the Age of Software Patents“, in which he proposed:

The basic principle of NATO is that an attack on any member is considered an attack on all members. In PATO it works like this–if any member of PATO is alleged with infringement of a software patent by a non-member, then that member may counter-sue the attacker based on infringement of any patent in the PATO cross-licensing pool, regardless of what member contributed it. Once a load of companies and patents are in the pool, this will be a deterrent equivalent to a couple thousand MIRVs in silos–odds are that any potential plaintiff will be more vulnerable to 10 or 20 PATO patents than the PATO member is to one patent from the aggressor. Perhaps the suit will just be dropped and the bad guy will decide to join PATO….

Since PATO is chartered to promote the free exchange and licensing of software patents, members do not seek revenue from their software patents–only mutual security. Thus, anybody can join PATO, even individual programmers who do not have a patent to contribute to the pool–they need only pay the nominal yearly dues and adhere to the treaty–that any software patents they are granted will go in the pool and that they will not sue any other PATO member for infringement of a software patent.

It’s been almost two decades, but perhaps PATO is an idea whose time has come. And, even if a collective effort fails, individual companies might do well to focus less on intellectual property capture and more on collecting the kinds of nuisance patents currently favored by trolls. After all, the best defense is the credible threat of a good offense.

Conclusion

Even if you hate software patents, you can’t afford to ignore them if you are in the software industry. And I’m well aware that not everyone shares my view of software patents. But I hope those who do find useful advice in the above discussion. I’d love to see the software industry move beyond this innovation tax.

A couple of years ago, I wrote a post entitled “Life, the Universe, and SEO” in which I considered Google’s relationship with the search engine optimization (SEO) industry. Specifically, I compared it to the relationship that Deep Thought, the computer in Douglas Adams’s Hitchhikers Guide to the Galaxy, has with the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons.

Interestingly, both SEO and union protests have been front-page news of late. I’ll focus on the former.

Three recent incidents brought mainstream attention to the SEO industry:

• Two weeks ago, Google head of web spam Matt Cutts told the New York Times that Google was engaging in a “corrective action” that penalized retailer J. C. Penney’s search results because the company had engaged in SEO practices that violated Google’s guidelines. For months before the action (which included the holiday season), J. C. Penney was performing exceptionally well in broad collection of Google searches, including such queries as [dresses], [bedding], [area rugs], [skinny jeans], [home decor], [comforter sets], [furniture], [tablecloths], and [grommet top curtains]. As I write this blog post, I do not see results from jcpenney.com on the first result page for any of these search queries.
• This past Thursday, online retailer Overstock.com reported to the Wall Street Journal that Google was penalizing them because of Overstock’s now discontinued practice of rewarding students and faculties with discounts in exchange for linking to Overstock pages from their university web pages. Before the penalty, these links were helping Overstock show up at the top of result sets for queries like [bunk beds] and [gift baskets]. As I write this blog post, I do not see results from overstock.com on the first result page for either of these search queries.
• That same day, Google announced, via an official blog post by Amit Singhal (Google’s head of core ranking) and Matt Cutts, a change that, according to their analysis, noticeably impacts 11.8% of of Google search queries. In their words: “This update is designed to reduce rankings for low-quality sites—sites which are low-value add for users, copy content from other websites or sites that are just not very useful. At the same time, it will provide better rankings for high-quality sites—sites with original content and information such as research, in-depth reports, thoughtful analysis and so on.”

Of course, Google is always working to improve search quality and stay at least one step ahead of those who attempt to reverse-engineer and game its ranking of results. But it’s quite unusual to see so much public discussion of ranking changes in such a short time period.

Granted, there is a growing chorus in the blogosphere bemoaning the decline of Google’s search quality. Much of it focused on “content farms” that seem to be the target of Google’s latest update. Perhaps Google’s new public assertiveness is a reaction to what it sees as unfair press. Indeed, Google’s recent public spat with Bing would be consistent with a more assertive PR stance.

But what I find most encouraging is the Google’s recent release of Chrome browser extension that allows users to create personal site blocklists that are reported to Google. Some may see this is as a reincarnation of SearchWiki, an ill-conceived and short-lived feature that allowed searchers to annotate and re-order results. But filtering out entire sites for all searches offers users a much greater return on investment than demoting individual results for specific searches.

Of course, I’d love to see user control taken much further. And I wonder if efforts like personal blocklists are the beginning of Amit offering me a more positive answer to the question I asked him back in 2008 about relevance approaches that relied on transparent design rather than obscurity.

I’m a realist: I recognize that many site owners are competing for users’ attention, that most users are lazy, and that Google wants to optimize search quality subject to these constraints. I also don’t think that anyone today threatens Google with the promise of better search quality (and yes, I’ve tried Blekko).

Perhaps the day is in sight when human-computer information retrieval (HCIR) offers a better alternative to the organization of web search results than the black-box ranking that fuels the SEO industry. But I’ve waiting for that long enough to not be holding my breath. Instead, I’m encouraged to see a growing recognition that today’s approaches are an endless game of Whac-A-Mole, and I’m delighted  that at least one of the improvements on the table takes a realistic approach to putting more power in the hands of users.

Heading to Punta Cana for a week. Feel free to keep writing great comments — will catch up when I get back!

Unless you’ve been living in a cone of silence, you’ve probably heard about the epic war of words between Google and Bing. But just in case, here’s a quick summary:

Amit Singhal, Google Fellow: “Microsoft’s Bing uses Google search results—and denies it“:

Bing is using some combination of:

• Internet Explorer 8, which can send data to Microsoft via its Suggested Sites feature
• the Bing Toolbar, which can send data via Microsoft’s Customer Experience Improvement Program

or possibly some other means to send data to Bing on what people search for on Google and the Google search results they click. Those results from Google are then more likely to show up on Bing. Put another way, some Bing results increasingly look like an incomplete, stale version of Google results—a cheap imitation.

Harry Shum, Corporate Vice President, Bing: “Thoughts on search quality“:

We use over 1,000 different signals and features in our ranking algorithm. A small piece of that is clickstream data we get from some of our customers, who opt-in to sharing anonymous data as they navigate the web in order to help us improve the experience for all users.

Yusuf Mehdi, Senior Vice President, Online Services Division, Bing: “Setting the record straight“:

Google engaged in a “honeypot” attack to trick Bing. In simple terms, Google’s “experiment” was rigged to manipulate Bing search results through a type of attack also known as “click fraud.” That’s right, the same type of attack employed by spammers on the web to trick consumers and produce bogus search results. What does all this cloak and dagger click fraud prove? Nothing anyone in the industry doesn’t already know. As we have said before and again in this post, we use click stream optionally provided by consumers in an anonymous fashion as one of 1,000 signals to try and determine whether a site might make sense to be in our index.

Matt Cutts, Head of Webspam, Google: “My thoughts on this week’s debate“:

Something I’ve heard smart people say is that this could be due to generalized clickstream processing rather than code that targets Google specifically. I’d love if Microsoft would clarify that, but at least one example has surfaced in which Microsoft was targeting Google’s urls specifically. The paper is titled Learning Phrase-Based Spelling Error Models from Clickthrough Data and here’s some of the relevant parts:

The clickthrough data of the second type consists of a set of query reformulation sessions extracted from 3 months of log files from a commercial Web browser [I assume this is Internet Explorer. --Matt] …. In our experiments, we “reverse-engineer” the parameters from the URLs of these [query formulation] sessions, and deduce how each search engine encodes both a query and the fact that a user arrived at a URL by clicking on the spelling suggestion of the query – an important indication that the spelling suggestion is desired. From these three months of query reformulation sessions, we extracted about 3 million query-correction pairs.”

This paper very much sounds like Microsoft reverse engineered which specific url parameters on Google corresponded to a spelling correction. Figure 1 of that paper looks like Microsoft used specific Google url parameters such as “&spell=1″ to extract spell corrections from Google. Targeting Google deliberately is quite different than using lots of clicks from different places.

Let me start by saying that these are very serious words from very serious people.

Amit and Matt, both of whom I know personally, are not just two of the most prominent Google employees — they have a deep personal investment in Google’s search quality. Amit is personally responsible for much of Google’s web search ranking algorithm, and Matt is surely the person whom spammers (and many SEO consultants) most love to hate. There is no question in my mind that the emotion both of them are expressing is sincere.

I haven’t met Harry or Yusuf, but I have no reason to doubt their own sincerity — especially since everything they are saying seems consistent with the facts — in fact, consistent with the substantive parts of Google’s allegations. Indeed, the facts don’t really seem to be in dispute. And more generally, I’ve met some of the folks who lead the Bing team (like Jan Pedersen), and, like Matt, I believe they are thoughtful and sincere and are devoted to building a great search engine of their own.

The debate is not about the facts. Rather, it’s about what is right and wrong. I will try to summarize the two sides’ position without editorializing.

Bing is claiming that:

• Users have a right to do as they please with their own clickthrough data, which includes data from Google search sessions.
• Bing toolbar users opted in to share this clickthrough data with Bing.
• By using this clickthrough data, Bing creates value for users.

Google is claiming that:

• Bing’s specific targeting of Google clickthrough data amounts to copying Google and is wrong.
• Bing toolbar users are not necessarily aware that they are complicit in this behavior.
• Bing is disingenuous in understating how much it benefits from Google as a signal.

What do I think?

I agree with Bing that users have the right to do as they please with clickthrough data. I’d think Google would agree too, given that Google wrote the sermon on “the meaning of open“:

Open information means that when we have information about users we use it to provide something that is valuable to them, we are transparent about what information we have about them, and we give them ultimate control over their information.

I agree with all of the three points I listed as Google’s claims except for the part that Bing’s behavior is wrong. It’s up to users if they want to help Bing compete with Google. Do users know that they’re doing so? Probably not. But would they stop doing so if they did? I doubt it. I can’t see why most users would have a dog in this fight — and in fact, it may be in users’ interest to help Bing be more competitive.

I do think Bing should be forthright about what it is doing — and how much this user-provided data from Google search sessions is contributing to its own quality improvements. Bing can, of course, keep this information secret, but I’d think that Bing would want to defend its reputation as an innovator — especially as the David in a David vs. Goliath fight.

But I also think that Google should be careful with its accusations. Accusing Bing of not being innovative is one thing, and that accusation, backed by concrete examples, is probably enough to score points. But implying that Google owns its users’ clickthrough data and that Bing has no right to solicit that data from users is another thing entirely.

I’m curious to hear what others here think. It’s been a while since I could freely express opinions about Google and Bing, so I’m delighted to have such a hot controversy to incite discussion. Because everyone enjoys a muddle puddle tweetle poodle beetle noodle bottle paddle battle!

Last October, a certain blogger said:

LinkedIn needs to implement some kind of concept extraction to provide a useful topic facet (something I’d also love to see for their regular people search). This is a challenging information extraction problem, especially for the open web, but I also know from experience that it is tractable within a domain. Given LinkedIn’s professional focus, I believe this is a problem they can and should tackle.

Shortly after writing that post, I interviewed at LinkedIn and met Pete Skomoroch, who showed me an early preview of the work his team was doing to make skills a facet for exploring the space of LinkedIn member profiles. That demo made a strong impression on me, giving me a taste of the great products LinkedIn’s data scientists were working on in the lab.

And now I’m delighted that everyone can try out the beta launch of LinkedIn Skills which was announced today at O’Reilly’s Strata 2011 conference on Big Data.

As Pete says in his blog post:

If you search for a particular skill, we’ll surface key people within that community, show you the top locations, related companies, relevant jobs, and groups where you can interact with like minded professionals.  You’ll also be able to explore similar skills and compare their growth relative to each other.

I encourage you to check it out — whether you’re looking for experts on Hadoop, cheese, or anything else! It’s a beta, so I’m sure you’ll find rough edges; but I hope it gives you a sense of how LinkedIn’s data can enable a incredibly powerful and useful exploratory search experience.

No forward-looking statements, except to say that it only gets better from here!

While my blog has always been and will always be a personal one, I do operate under certain constraints as someone whose subject matter relates strongly to his professional interests. I deeply appreciate how long-time readers have respected the balancing act I sometimes have to make as both an independent individual and an employee.

Right now, that means I must respect the conditions of my employer’s quiet period — and I will do so very conservatively (e.g., no Playboy interviews). I apologize if the content of this blog suffers in the interim, but I hope you understand my need to be cautious.

And one last question…are looking for an exciting internship opportunity this summer? Then you’ve come to the right place at the right time: LinkedIn is looking for a few good interns for summer 2011! You can find more details here or go directly to the application form.

If you are interested, I encourage you to act quickly, since we are already interviewing candidates.

“If a man hasn’t discovered something that he will die for, he isn’t fit to live.”

Martin Luther King, Jr. said these words at a speech in Detroit on June 23, 1963. Less than five years later, he died for the cause to which he devoted his life: the advancement of civil rights in the United States and around the world through civil disobedience and other nonviolent resistance.

Today, as Americans commemorate Dr. King’s birthday, there are many ways we can honor his memory and build on his legacy. As much as King advanced the civil rights movement, there is still much to be done to fulfill his dream.

But I’d like to go back to the quote from his speech in Detroit. King’s words reveal a truth even deeper than his struggle for civil rights. They demand that we approach life with passion, that we live to do something more than pass the time.

In the face of pressing day-to-day responsibilities, it is easy to fall into a reactive rhythm, doing what we have to do and then using what time is left to escape into oblivion. For many of us, passion may feel like a nice-to-have, something to think about after we’ve cleared out our queues and gotten a full night of sleep — only to wake up and find that the queue is full again. It is easy to go through life like Sisyphus, sweating profusely as we roll our boulders but lacking the intellectual ambition to question why we make those efforts.

Today, the least we can do to honor the memory of Martin Luther King, Jr. is to reflect on his personal passion to leave the world better than he found it. Hopefully none of us will ever have to make the sacrifice that he made to realize his dream. But if we do not dare to dream at all — if we are not passionate and ambitious about what do — then, indeed, we are not fit to live.

Dare to dream — and live to make that dream a reality.

But I have personal reasons to be interested in Quora. One of their lead engineers, Albert Sheu, was a star intern of mine at Endeca. And Quora raises lots of interesting questions about search, user experience, knowledge management, and online reputation. How could I resist?

I see three potential reasons to use Quora:

1. Objective question answering.
2. Subjective question answering.
3. Community participation.

Let’s consider how Quora fares today on each of these, and where it might go.

1. Objective question answering.

When I blogged about Quora early last year, I said that “I don’t see Quora as a knowledge base of first resort–except possibly to learn more about software startups.” Despite Quora’s recently growth surge, I am not ready to change my answer significantly — I find that Quora’s topics are pretty sparse when I stray from its Silicon Valley focus.

Within that focus, Quora is nailing it. For example, I was curious to learn whether someone who signed a non-compete agreement outside of California was still subject to it if he or she moved to California, where such contracts are legally unenforceable. Not surprisingly, non-compete agreements are a topic on Quora, and I quickly found a useful answer from a lawyer.

But for most objective questions, I’m still turning to Google and Wikipedia — or to Twitter if both of those fail and I am willing to ask a favor of my followers (who kick ass!). Sometimes Google will take me to Quora, but I can’t imagine Quora will succeed through this flow in the long term.

2. Subjective question answering.

I see subjective question answering as Quora’s strongest suit. A good subjective question on Quora — often a “why” question — generates a diverse collection of interesting and informed perspectives. A couple of good example are “Why did Google Wave fail to get significant user adoption?” and “What is lacking in social networking now?“.

Again, these questions are well within the Silicon Valley focus, but I could see Quora extending this value proposition to other verticals if it can grow the communities successfully. And I certainly don’t see myself going to Google or even Twitter to get useful answers to subjective questions. The closest is Topsy, and Quora has the advantage of being explicitly organized around questions and topics.

3. Community participation.

Is Quora a question answering site or a social network? Quora users and employees have tried to answer that question (on Quora, natch), but I’m not sure Quora’s converged enough for anyone to know. What is clear is that Quora emphasizes conversation, making it more like a blog or wiki than an answers site.

Conversation certainly engages its participants. But it also raises the cost of participation. One of the things I love about Google is that it gives me information without unnecessary overhead. When I want conversation, I go to social venues like Twitter.

Perhaps Quora can be both a question answering site and a social network. But I suspect it will need to choose. Most people don’t have the time or patience to participate in additional communities, so question answering is the easier sell to a mass audience. But the participation is what makes Quora especially distinctive today. Perhaps it’s a question of quality vs. quantity.

So, quo vadis, Quora? I suppose I’ll have to check Quora (or Cwora) to find the answers.

I hope everyone I’ve invited is enjoying Quora. But I also hope you take it upon yourselves to circulate more invitations to those who want them. Any Quora user can send out invites — that’s how these viral sites work. If you’re still looking for an invite, I urge you to use Twitter or some other broadcast mechanism to request it. As of today, I will stop responding to Quora invite requests through my blog or email, and I will also delete comments requesting them. I am sorry if this is a bit harsh, but I hope folks understand.

Please use the comments here to provide feedback. Bharath is eager to improve his product, and I’m eager to improve the experience for all of you!

The first is that I’ve eliminated the use of categories on posts. I found that I was categorizing almost all posts as “general”, and that there was almost no value in maintaining such a low-entropy field. Yes, I’m well aware of the irony that, despite being an advocate of faceted search, I’m not providing any meta-data to annotate my posts. But I take what I believe to be a user-driven approach, and I can see from my logs that readers primarily visit to read my most recent posts. For those who like to browse, I encourage you to take advantage of the related posts feature, which is powered by the Yet Another Related Posts Plugin.

The second is that I’ve retired The Noisy Community. Two years ago, I created this directory of regular readers and commenters in order to foster a sense of community. I believe it was a success, but that it has outlived its usefulness. Again based on my logs, I can see that it has been neglected for a while. So, rather than continuing to invest in maintaining it manually, I have given it an honorable discharge.

My apologies to any readers whom I’ve offended with these changes. As always, I encourage you to make suggestions — especially if you’re willing to help implement them!

But perhaps you’d like to make a career out your jones for big data. Luckily for you, some of the hottest companies around are hiring data scientists!

Of course, those jobs aren’t for everyone. To get an idea of the necessary qualifications, I suggest you read the answers on Quora for “How do I become a data scientist?” to get an idea of the requisite math and computer science skills. I’m also a fan of Hilary Mason‘s definition which was cited in Ryan Kim’s “Wanted: Data Scientists to Turn Information Into Gold“: a data scientist is someone who can obtain, scrub, explore, model and interpret data, blending hacking, statistics and machine learning. You can see Hilary’s full explanation in a blog post she co-authored with Chris Wiggins, entitled “A Taxonomy of Data Science“.

If the qualifications haven’t scared you off, then it’s just a question of where you can best apply your data scientist skills. The good news is that there are a lot of different ways to make a career out of working with big data. Here are some suggestions for what to work on. I apologize in advance for taking a US-centric perspective — if you’re outside the US, I can only hope that the examples have local analogs.

1) Web search.

Google, Yahoo, and Bing all collect an enormous amount of data from people’s web search activity. Google is, of course, the 800-pound gorilla, but don’t dismiss the others — even a single-digit market share is enough to derive extremely valuable insights from user activity. And, since every major search engine makes the bulk of its revenue from advertising, they all present the big-data challenges associated with computational advertising. Search is, in my view, the web’s killer app, so you can’t go wrong working on it. But temper your expectations — despite heroic efforts from various parties, it seems difficult to deliver revolutionary improvements to this field.

2) Social networking.

Here the biggest players are Facebook and Twitter, but you can find a more comprehensive list on Wikipedia. Many consider LinkedIn to be a social network, but I’ll take the liberty to discuss it in its own section. Social networks attract an outsized share of users’ attention: Facebook alone accounts for a quarter of US page views on the web! All of this user activity means a lot of data to crunch, so it’s not surprising that LinkedIn, Facebook, and Twitter are recognized as having the best data science teams. How much you’ll enjoy working at these companies will in part reflect the value (and values) you perceive in their offerings, but they are all playgrounds for data scientists.

3) Electronic commerce.

While ad-supported web search may be the killer app of the web, what opens up people’s wallets is e-commerce. Led by Amazon and eBay, e-commerce sites deserve much of the credit for turning the web from an esoteric research project into a mainstream staple. And, like their offline counterparts, e-commerce sites generate vast amounts of data from how users view and purchase products. This data drives user recommendations, merchandising campaigns, pricing strategy, and much more. If you’d like to pursue data-driven capitalism, then e-commerce may be for you. A word of caution: if you are one of a crowd of merchants selling the same products as everyone else (as opposed to a site like Etsy selling unique products), make sure you have a sustainable competitive advantage. Data science is necessary for success in e-commerce, but it may not be sufficient.

4) Digital content.

Whether its books, music, video, or apps, the long-prophesied digital convergence has arrived: almost every newly created piece of digital content is now distributed in electronic form. Here the biggest players are Amazon, Apple, and Google (particularly its YouTube subsidiary), but there is still a lot of flux as new hardware, software, and business models compete for dominance. Digital content poses two daunting challenges: the volume of published content far exceeds people’s available attention, and digital media products are experience goods than people can only evaluate after consuming them. For both of these reasons, the digital content industry depends on data scientists to help people find and discover what they like. The catch: from its advent, the digital content industry has struggled with unauthorized distribution (aka piracy), and the results of this struggle will determine which business models are viable.

5) Finance.

Money, money, money. Working in finance has always been a data-intensive business, but advances in technology have only increased the industry’s reliance on data scientists. Algorithmic trading — and high-frequency trading in particular — mean that those who can most effectively and efficiently mine financial data can derive enormous financial benefits. Finance isn’t for everyone — the hours are long, the stress is high, and the compensation is highly variable. That said, the financial upside can be quite compelling, and some even enjoy the lifestyle.

6) Public sector.

Given the libertarian leanings of the software industry, the public sector might not seem like an obvious career choice. But some of the largest repositories of data reside there–from public repositories like census data to highly classified repositories restricted to the TLAs. Better understanding of this data can improve public policy, national security, and much more. Not everyone has the temperament to deal with government bureaucracy, but those who do have the opportunity to turn big data into big public good.

7) LinkedIn.

OK, I’m being self-serving, but after all this is my blog! LinkedIn is widely recognized as being one of the top data science teams on the planet. But LinkedIn has more than just talent — it has what Pete Warden of ReadWriteWeb described in “Secrets of the LinkedIn Data Scientists” as “detailed information on millions of people who are motivated to keep their profiles up-to-date, collect a rich network of connections and have a strong desire from their users for more tools to help them in their professional lives.” Indeed, I don’t know of anyone who has a dataset that competes with the combined quantity, quality, and utility of LinkedIn’s data. Moreover, working as a data scientist at LinkedIn means helping make people more professionally successful by connecting the to opportunities, information, and of course other people. It’s a wonderful way to create value, and it doesn’t hurt to do so in the context of a profitable, rapidly growing company.

And LinkedIn recognizes the extraordinary value of data science. Don’t take my word for it — listen to LinkedIn CEO Jeff Weiner’s interview at the 2010 Web 2.0 Summit:

To wrap up, data science is more than just an opportunity to have fun and make the world a better place — it might even be how you make an honest living!

I began this year as a Noogler and leave it as a Xoogler. I hope I left Google better than I found it — I’m certainly proud of the improvements my team made to the quality of local authority pages. I also tried to infuse Google with some of the scrappy start-up culture I’d picked up at Endeca, particularly focusing on the hiring process. In information retrieval terms, I’d say that Google’s hiring process does extremely well when it comes to precision, but could use improvement in the areas of recall and efficiency. Still, I’m impressed at how well Google has maintained its quality standards as the company has grown. Finally, I couldn’t help being an extrovert: I developed warm relationships with the lead bloggers covering local search, including Andrew Shotland, David Mihm, Gib Olander, Greg Sterling, and Mike Blumenthal. Indeed, when I announced my departure, Mike wrote a really nice post about the friendship we cultivated over the past year. I hope that he continues to have such relationships with my former co-workers.

Looking back at what was on my mind when this year began, I had lots of questions around exploratory, mobile, real-time, social/collaborative search. I also wondered whether it was possible to offer more transparency in relevance ranking without losing ground in the battle against spam and black-hat SEO.

I’m as bullish as ever on the value of exploratory search:  part of why I joined LinkedIn is that a significant fraction of the site’s value comes from supporting users’ exploratory search needs. I also published a position paper at the SIGIR 2010 Workshop on Simulation of Interaction proposing the use of query performance prediction to model the fidelity of communication between user and system, thus helping HCIR researchers to simulate query refinement with standard test collections. And of course exploratory search was a major theme at the HCIR 2010 workshop, not only providing the basis for the first HCIR Challenge, but even extending to new territory with Max Wilson and David Elsweiler’s work on casual leisure searching.

As for mobile search, I’d say that 2010 has been the year of “mobile first“. Thanks to a generous gift from my former employer, I’ve become a regular user of the mobile web–and of search in particular. To my surprise, the communication bottleneck has not been screen real estate, but rather the difficulty of entering text. And innovative approaches like voice search and Swype go a long way to mitigate that difficulty.

On to real-time search. Not surprisingly, my favorite innovation in this space is LinkedIn Signal, which offers exploratory search for Twitter. I still struggle to find use cases that emphasize the “real-time” aspect of Twitter and other microblogging services, but I am convinced that the path to utility lies in tools that support organization, analysis, and exploration.

On the social/collaborative front, I’m happy to work for a company whose charter includes “supporting mediated search by linking people to people, rather than directly to information”. While the biggest event in this space in 2010 was Facebook’s introduction of the Like button, I’m not convinced that “likes” have supplanted links. I’m still looking to niche players like Topsy and Blekko to push innovation in this space.

Speaking of Blekko, they’ve made an impressive attempt to increase the transparency of relevance ranking. But, as I blogged earlier this year, I think that, at least for the time being, Google is making the right decision to keep some of its details secret. Now that web search is essentially a duopoly (at least in the US), I believe the real test of the value of transparency to users will be whether one of the two parties employs it as competitive differentiator.

What’s in store for 2011? LinkedIn CEO Jeff Weiner has a vision of using data science to provide a “Pandora for people“, and that’s a vision I’m eager to help realize. Not surprisingly, when I blogged in 2008 about where Google wasn’t good enough, two of the four areas I cited were finding jobs and find employees. Even then I recognized that LinkedIn was the best at both. But LinkedIn can be so much more, and I am looking forward to working with an incredible team and incredible data on a delightful set of information science challenges.

Happy New Year! I hope that 2011 brings you great answers — and great questions!

I started off by considering a model where each individual in an organization leaks a particular piece of sensitive information with a constant probability p, and where acts of leakage are independent and identically distributed events. Now let’s consider what value of p leads to a 99% probability of leakage in an organization of n = 20,000 people. It’s less than 1/4000. In other words, even if each person in an organization can keep a secret with 99.98% reliability, almost all secrets will be leaked.

Using this same value of p with n = 900 (roughly the size of my current employer) yields less than a 20% chance of leakage — certainly not a zero probability, but much closer to zero than to one. And at n = 90 — the upper end of what I’d consider a startup — the probability of leakage drops to 2%. Based on this crude analysis, the ability to keep secrets drops very rapidly as organizations enjoy the growth that comes with success.

Moreover, p is likely to be positively correlated to n — that is, individuals in larger organizations are more likely to leak sensitive information. Many people in larger organizations have less actual and perceived stake in the organization’s success, than those in smaller ones. Also, it is difficult to sustain grueling hiring standards — particularly cultural ones — as an organization grows.

So what is an organization to do? If the above model is even close to accurate, then I can see four options:

1) Don’t grow.

Yes, I’m serious. Not every idea inspires a billion-dollar business, and not every company should grow beyond a hundred people. Growth has costs that offset its benefits, and the inability to keep secrets may be a significant cost for organizations whose competitive advantage depends on proprietary intellectual property. The largest hedge funds each have about 1,000 employees, and most are much smaller. Secrecy is not the only consideration, but it’s certainly a consideration.

2) Share less with your employees.

If you can’t reduce p, you can at least reduce n by sharing secrets less widely. Traditional organizations only share sensitive information within a tight inner circle. Even Google, known for sharing almost everything with its employees, keeps tighter control over the details of search result ranking. This approach, however, comes at a cost: it signals to employees that they cannot be trusted. Moreover, if employees discover secret information through rumor, they may feel less responsible for maintaining secrecy than if they had been entrusted with that information.

3) Investigate leaks and punish leakers.

Some organizations succeed better than others at rooting out leakers and punishing them. In economic terms, it makes sense to discourage undesirable behavior through strong disincentives. Note, however, that leakers rarely gain anything tangible in exchange for their leaks and indeed are often acting irrationally in strictly economic terms. People in general have been known to act irrationally. So I’d caution against any approach that assumes human rationality. A better approach may be to detect or prevent of leaks through technology (e.g., packet analyzers), but see the previous comment about making employees feel they cannot be trusted.

4) Keep fewer secrets.

A prominent CEO recently said “If you have something that you don’t want anyone to know, maybe you shouldn’t be doing it in the first place“. Yes, I’m taking the quotation out of context, but I’d like to offer a variant: if your organization’s success depends on something that you don’t want anyone to know, maybe you should reconsider your business model. Less glibly, you should avoid unnecessary dependence on secrecy, and you should avoid labeling all corporate information as secret, since that desensitizing employees to the risks of disclosure.

Conclusion? As Ben Franklin said, “Three may keep a secret, if two of them are dead.” Organizations can and do manage to keep secrets. But it’s hard to fight human nature, and better not to rely on winning that fight.

CIKM 2011 is nearly a year away, but I wanted to give folks a heads up about the Industry Event there that I am organizing with Tony Russell-Rose. These events have become an an increasingly important part of the annual CIKM and SIGIR conferences, and I believe they are helping to bridge the gap between scholarship and practice. When I organized the SIGIR 2009 Industry Event, it was almost too popular — I felt bad for the parallel research presentations that had to compete with Matt Cutts and danah boyd for attendees!

But not so bad that I wouldn’t do it again! We have an outstanding line-up of invited talks for the CIKM 2011 Industry Event, featuring:

• Stephen Robertson (Microsoft Research)
• John Giannandrea (Google)
• Jeff Hammerbacher (Cloudera)
• Peter Jackson (Thomson Reuters)

For those not familiar with industry luminaries, that list includes one of the world’s most prominent information retrieval researchers, the founder of Metaweb (which created Freebase), the person who build Facebook’s data team (which developed Hive and Cassandra), and one of the leading industrial researchers on natural language processing. To borrow a sports metaphor, these were our first-round draft picks, and we are delighted that they all agreed to participate.

And those are just the keynotes! We’re also going to put out a call for participation soon, so watch this space!

Having just started my new life as a citizen of Silicon Valley, I’ve quickly comprehended how it is the perfect birthplace for LinkedIn. Every introduction has been an exercise of triadic closure. Indeed, while most people know that the Bay Area is the world’s leading hub for technology startups, perhaps not everyone realizes that the foundation for this environment is the professional network that binds it. I’ve only been here for a week, and yet my world seems smaller by the day as I keep discovering new connections among my colleagues. It’s a lot of fun, if a bit overwhelming!

And fun but overwhelming is a great way to describe LinkedIn itself. It’s only been a few days since I updated my profile, but I already feel immersed in LinkedIn’s vibrant culture. I sit in an open office, surrounded by people I work with — data scientists, software engineers, product managers, designers, and more. And I’m already interviewing folks I might be working with soon — in a company growing as quickly as LinkedIn, it is everyone’s job to grow the team. I’ve joked to friends that moving west gave me three more hours to get work done — but I’m using them all and they’re not enough!

But despite this explosive growth, LinkedIn’s vision is shared and tight. We all know that our goal is to connect the world’s professionals to make them more productive and successful. Having such a clear-cut mission enables us to directly relate all of our efforts and ambitions to the concrete value they create. It’s a great feeling, and it helps me keep my sanity as I observe the size of my ever-increasing to-do list.

To say that I’m still adjusting is an understatement. I haven’t made a change like this is over a decade, and this adventure feels even more immersive. But a big difference between now and 1999 is that I arrive in my new world with a network of people there to welcome me. I have LinkedIn to thank for helping me develop that network, and it’s great to finally have the opportunity to give back.

Today is my last day at Google. I have enjoyed an incredible year there, during which I’ve had the privilege to work with some of the smartest engineers on the planet. Working at Google taught me how much impact a handful of dedicated people can have on the lives of billions of users. Not that long ago, I compared Google to McDonald’s. Having spent time on the inside, I can attest that Google is a marvel of scale orchestration. Moreover, the Google New York office represents an impressive concentration of Google’s talent in the greatest city of the world.

But I am leaving Google to pursue the opportunity of a lifetime. On Monday, I will start a new chapter of my life. I am joining the data scientist team at LinkedIn, where I’ll be working with DJ Patil and his world-class team to build products and discover insights from a data collection that I have coveted for years. I’ll get to work with folks like Pete Skomoroch and Monica Rogati. And I’ll get to tackle challenges in my favorite areas of computer science: information extraction, matching, recommendation, social network analysis, and network visualization. Not to mention working with one of the largest faceted search deployments on the web!

It was an agonizing decision to leave Google and New York City. But, when LinkedIn reached out to me a couple of months ago, I was reminded of a fateful email from Steve Papa in July 1999 that led me to pack a bag two months later and begin the adventure that is now Endeca. LinkedIn is hardly a startup — it has over 600 employees and over 80 million members. But I see boundless opportunities to create new value from the great data and talent that LinkedIn has assembled. So, when I received that note from LinkedIn, I didn’t really have a choice.

This Monday, I begin a new adventure. Data, here I come!

Consider information retrieval. Perhaps I’m dating myself, but an undergraduate computer science major, I hardly imagined that information retrieval would have much significance outside of academia. Sure, there were commercial IR systems being built in the 1980s, but it wasn’t until the late 1990s that web search brought IR to the mainstream. Today, it’s hard to imagine studying computer science without learning about IR. Sure, my career makes me a tad biased, but it is undeniable that information retrieval is one of the defining problems of our generation.

And then there are social networks. When I studied graph drawing in the 1990s, the canonical example of a social network was “Six Degrees of Kevin Bacon“. Sure, many of my peers would talk about their Erdős numbers (they were more discreet about their placement in the Tarjan graph), but the study of social networks was surely an academic pursuit. Who would imagine that, barely a decade later, a movie entitled The Social Network would be a blockbuster movie grossing $175M? Leaving aside Hollywood, social networks have become a significant part of our daily lives. Not only do Facebook, Twitter, and LinkedIn account for a large fraction of our time online, but they also affect our offline personal and professional lives.

From childhood, I’ve been interested in mathematics, computer science, and psychology. Living in an age of information retrieval and social networks means that I can apply these interests in my daily work. Today I give thanks for being born at the right place and right time, blessed with a lifetime of interesting and practical problems to solve. Happy Thanksgiving to all, and enjoy the leftovers!

I excerpt the following example from section 16.2, which the authors in turn  borrow from Lisa Anderson and Charles Holt:

The experimenter puts an urn at the front of the room with three marbles in it; she announces that there is a 50% chance that the urn contains two red marbles and one blue marble, and  a 50% chance that the urn contains two blue marbles and one red marble…one by one, each student comes to the front of the room and draws a marble from the urn; he looks at the color and then places it back in the urn without showing it to the rest of the class. The student then guesses whether the urn is majority-red or majority-blue and publicly announces this guess to the class.

Let’s simulate how a set of rational students would perform in this experiment.

The first student has it easy: if he selects a blue marble, he guesses blue; if he selects a red marble, he guesses red. Either way, his guess publicly discloses the first marble’s color.

This the second student knows exactly the colors of the first two selected marbles. If he selects the same color as the first student, he will make the same guess.  If, however, the second student selects a red marble, he has no reason to prefer one color over the other. Let’s assume that, when the odds are 50/50, an indifferent student breaks symmetry by selecting the color in his hand. That way, we guarantee that the second student discloses the color of the marble he selects.

Things get interesting with the third student’s selection. What happens if the first two students have both guessed red, but the third student selects a blue marble? Rationally, the third student will guess red, since he knows that two of the first three selected marbles were red. In fact, if the first two students select red marbles, *every* subsequent student will ignore his own selection and guess red. Of course, analogous reasoning applies if we reverse the colors.

Generalizing from this case, we can see that the sequence guesses locks in on a single color as soon as two consecutive students agree. I leave it as an exercise to the reader to determine that, if the urn is majority-red, there is a 16/21 probability that the sequence will converge to red and a 5/21 probability that it will converge to blue.

A 5/21 probability of arriving at the wrong answer may not seem so bad. But imagine if you could see the actual marbles sampled and not just the guesses (i.e., each student provides an independent signal). The law of large numbers kicks in quickly, and the probability of the sample majority color being different from the true majority converges to 0.

This example of an information cascade is unrealistically simple, but is eerily suggestive of the way many sequential decision processes work. I hope we all see it as a cautionary tale. The wisdom of the crowd breaks down when we throw away the independent signals of its participants.

Like many interface design principles, the principle of least surprise reflects the premise that software applications exist to be useful. In utility-oriented applications, surprise means distraction and delay — negatives that good designers work to avoid.

But we increasingly see applications whose main value to the user is not utility, but entertainment. Indeed, a recent Nielsen report claims that the top two online activities for Americans are social networks / blogs and games. I take the report with a grain of salt, but it seems safe to argue that people have come to expect the internet to be at least as fun as it is useful.

Even search, which would seem to be the poster child for the utility of online services, is being pressed into the service of entertainment. Max Wilson and David Elsweiler argued as much in their HCIR 2010 presentation about “casual leisure searching“. They mined Twitter to analyze a variety of scenarios where search isn’t about the use finding something, but rather about enjoying the experience. Indeed, their controversial definition of search is broad enough to include the possibility that the user does not have an information need.

Like the businessman in Antoine de St. Exupery’s Le Petit Prince, I’ve long felt that, as “un homme sérieux”, my job is delivering utility to users. Users already have lots of ways to waste time; I focus on making their productivity-oriented time more effective and efficient. I’m glad there are folks who devote their lives to making the rest of us have more fun (especially all the computer scientists who left academia for Pixar), but entertainment simply isn’t a vocation for me.

However, I’ve been coming around to the realization that fun and utility are not mutually exclusive. For example, news serves the utilitarian ideal of informing the citizenry, but many (most?) of us read news as a pleasant way to pass the time. Social networks are another example serving a similar function–perhaps with a balance that is more toward the entertainment of the spectrum but still providing genuine social utility.

A common feature of both of these examples is that users regularly return to the same site expecting the unexpected. The transient nature of news and social news feeds promises an endless supply of fresh content, produced more quickly than users can consume it. This situation is in stark contrast to those of typical web search queries, for which the results are expected to be largely static. Indeed, we may set up alerts to inform us of novel search results, but we are unlikely to regularly visit a bookmarked search results page the way we regularly visit a news or social network site.

Is novelty the only source of surprise? Novelty certainly helps, but it is not a necessity. An alternative source is randomness. I’m known people to use Wikipedia’s “random article” feature. But a more plausible place to introduce randomness is in recommendations — whether for products or content. Since recommendations are good guesses at best, a bit of randomness can help ensure that the guesses are interesting. Indeed, a SIGIR 2010 paper by Neal Lathia, Stephen Hailes, Licia Capra, and Xavier Amatriain on “Temporal Diversity in Recommender Systems” explored the use or randomness to induce diversity in recommendations and arrived at the conclusion that people don’t like being recommended the same things over and over again.

Can we generalize from these examples? I think so. For utility-oriented information needs, it is important to provide users with accurate, predictable, and efficient tools. But we can’t dismiss everything else as frivolous. Sometimes we just need to offer our users a little bit of surprise to keep it interesting.

Or, as Mary Poppins tells us: “In every job that must be done, there is an element of fun. You find the fun, and – SNAP – the job’s a game!”

From an information scientist’s perspective, these are two distinct problems to solve in the information seeking process: establishing the user’s information need (query elaboration) and retrieving relevant information (information retrieval).

When open-domain search engines (i.e., web search engines) went mainstream in the late 1990s, they did so by glossing over the problem of query elaboration and focusing almost entirely on information retrieval. More precisely, they addressed the query elaboration problem by requiring users to provide reasonable queries and search engines to infer information needs from those queries. In recent years, there has been more explicit support for query elaboration–most notably in the form of type-ahead query suggestions (e.g., Google Instant). There have also been a variety of efforts to offer related queries as refinements.

But even with such support, query elaboration typically yields an informal, free-text string. All vocabularies have their flaws, but search engines compound the inherent imprecision of language by not even trying to guide users to a common standard. At best, query suggestion nudges users towards more popular–and hopefully more effective–queries.

In contrast, consider closed-domain search engines that operate on curated collections, e.g., the catalog search for an ecommerce site. These search engines often provide users with the opportunity to express precise queries, e.g., black digital cameras for under $250. Moreover, well-designed sites offer users faceted search interfaces that support progressive query elaboration through guided refinements.

Many (though not all) closed-domain search engines have an advantage over their open-domain counterparts: they can rely on manually curated metadata. The scale and heterogeneity of the open web defies human curation. Perhaps we’ll reach a point when automatic information extraction offers quality competitive with curation, but we’re not there yet. Indeed, the lack of good, automatically generated metadata has been cited as the top challenge facing those who would implement faceted search for the open web.

What can we do in the mean time? Here is a simple idea: use a closed-domain search engine do guide users to precise queries, and then apply the resulting queries to the open web. In other words mash up the closed and open collections.

Of course, this is easier said that done. It is not at all clear if or how we can apply a query like “black digital cameras for under $250″ to a collection that is not annotated with the necessary metadata. But we can certainly try. And our ability to perform information retrieval from structured queries will improve over time–in fact, it may even improve more quickly if we can start to assume that users are being guided to precise, unambiguous queries.

Even though result quality would be variable, such an approach would at least eliminate a source of uncertainty in the information seeking process: the user would be certain of having a query that accurately represented his or her information need. That is no small victory!

I fear, however, that users might not respond positively to such an interface. Given the certainty that a query accurately represents his or her information need, a user is likely to have higher expectations of result quality than without that certainty. Retrieval errors are harder to forgive when the query elaboration process eliminates almost any chance of misunderstanding. Even if the results were more accurate, they might not be accurate enough to satisfy user expectations.

As an HCIR evangelist, I am saddened by this prospect. Reducing uncertainty in any part of the information seeking process seems like it should always be a good thing for the user. I’m curious to hear what folks here think of this idea.

Last week, I had the pleasure of talking with CMU professor George Loewenstein, one of the top researchers in the area of behavioral economics. I mentioned my idea of using prediction markets to address the weaknesses of online review systems and reputation systems, and he offered two insightful pointers.

The first pointer was to the notion of pluralistic ignorance. As summarized on Wikipedia:

In social psychology, pluralistic ignorance, a term coined by Daniel Katz and Floyd H. Allport in 1931, describes “a situation where a majority of group members privately reject a norm, but assume (incorrectly) that most others accept it…It is, in Krech and Crutchfield’s (1948, pp. 388–89) words, the situation where ‘no one believes, but everyone thinks that everyone believes’”. This, in turn, provides support for a norm that may be, in fact, disliked by most people.

It had not occurred to me that pluralistic ignorance could wreak havoc on the prediction market approach I proposed. Specifically, there is a risk that, even though the majority participants in the market hold a particular opinion, they suppress their individual opinions and instead vote based on mistaken assumptions about the collective opinion of others. Ironically, these participants are pursuing an optimal strategy, given their pluralistic ignorance. Yet the results of such a market would not necessarily reflect the true collective opinion of participants. Clearly there is a need to incorporate people’s true opinions into the equation, and not just their beliefs about others’ opinions.

Which leads me to the second resource to which Loewenstein pointed me: a paper by fellow behavioral economist and MIT professor Drazen Prelec entitled “A Bayesian Truth Serum for Subjective Data“. As per the abstract:

Subjective judgments, an essential information source for science and policy, are problematic because there are no public criteria for assessing judgmental truthfulness. I present a scoring method for eliciting truthful subjective data in situations where objective truth is unknowable. The method assigns high scores not to the most common answers but to the answers that are more common than collectively predicted, with predictions drawn from the same population. This simple adjustment in the scoring criterion removes all bias in favor of consensus: Truthful answers maximize expected score even for respondents who believe that their answer represents a minority view.

Most of the paper is devoted to proving, subject to a few assumptions, that the optimal strategy for players in this game is to tell what they believe to be the truth–that is, the truth-telling strategy is the optimal Bayesian Nash equilibrium for all players.

The assumptions are as follows:

1. The sample of respondents is sufficiently large that a single answer cannot appreciably affect the overall results.
2. Respondents believe that others sharing their opinion will draw the same inferences about population frequencies.
3. All players assume that other players are responding truthfully–which follows if they are rational players.

Prelec sums up his results as follows:

In the absence of reality checks, it is tempting to grant special status to the prevailing consensus. The benefit of explicit scoring is precisely to counteract informal pressures to agree (or perhaps to “stand out” and disagree). Indeed, the mere existence of a truth-inducing scoring system provides methodological reassurance for social science, showing that subjective data can, if needed, be elicited by means of a process that is neither faith-based (“all answers are equally good”) nor biased against the exceptional view.

Unfortunately, I don’t think that Prelec’s assumptions hold for most online review systems and reputation systems. In typical applications (e.g., product and service reviews on sites like Amazon and Yelp), the input is too sparse to even approximate the first assumption, and the other two assumptions probably ascribe too much rationality to the participants.

Still, Bayesian truth serum is a step in the right direction, and perhaps the approach (or some simple variant of it) applies to a useful subset of real-world prediction scenarios. Certainly it gives me hope that we’ll succeed in the quest to mine “subjective truth” from crowds.

I like Twitter. Yes, I know that a lot of its content is noise. But I’ve found Twitter to be a useful professional tool for both publishing and consuming information. Publishing to Twitter is the easy part: I publish links to my blog posts and occasionally engage in public conversations.

Consuming information from Twitter is more of a challenge. I follow 100 people, which is about the limit of my attention budget. I use saved searches to track long-term interests (much as I use web and news alerts), and I perform ad hoc searches when I am interested in finding out what people are saying about a particular topic.

But Twitter search is not a great fit for analysis or exploration–unless you count trending topics as analysis. Originally, the search results were simply the tweets that contained the matching tweets in order of recency. The current system sometimes promotes a few “top tweets” to the top of the results. Still, if you’d like to get a summary view, slice and dice the results, or perform any other sort of HCIR task, you’re out of luck.

Until now.

The LinkedIn Search, Network, and Analytics team–the same folks that built LinkedIn’s faceted search system and developed open-source search tools Zoie and Bobo–just introduced a service called Signal that is squarely aimed at folks like me who use Twitter as a professional tool. It is still in its infancy (in private beta, in fact), but I think it has the potential to dramatically change how people like me use Twitter. You can learn more about its architecture and implementation details here.

Signal joins the often cacophonous Twitter stream to the high-quality structured data that LinkedIn knows about its own users. For example, when I post a tweet, LinkedIn knows that I am in the software industry, work at Google, and live in New York. LinkedIn can only make this connection for people who include Twitter ids in their LinkedIn profiles, but that’s a substantial and growing population.

Signal then lets you use this structured information to satisfy analytic and exploratory information needs. For example, I can see which companies’ employees are tweeting about software patents (top two are Google and Red Hat).

Or compare what Microsoft employees are saying about Android…

…to what Google employees are saying about Android.

As you can see on the right-hand side, Signal also mines shared links to identify popular ones relative to given search–and allows you to see who has shared a particular link. This functionality is similar to Topsy, but with the advantage of allowing structured searches. Like Topsy, it wrangles the mass of retweeted links into a useful and user-friendly summary.

Signal is still very much in beta. An amusing bug that I encountered earlier today was that, due to some legacy issues in how Linkedin standardized institution names, the system decided that I was an alumnus of the Longy School of Music rather than of MIT. Fortunately, that’s fixed now (thanks, John!)–I love karaoke, but I’m not ready to quit my day job!

Also, Signal only exposes a handful of LinkedIn’s facets, which limits the breadth of analysis and exploration. I’d love to see it add a past company facet, making it possible to drill down into what a company’s ex-employees are saying about a particular topic (e.g., their ex-employer).

Finally, while Signal offers Twitter hashtags as a facet, these are hardly a substitute for a topic facet. In order to provide such a facet, LinkedIn needs to implement some kind of concept extraction to provide a useful topic facet (something I’d also love to see for their regular people search). This is a challenging information extraction problem, especially for the open web, but I also know from experience that it is tractable within a domain. Given LinkedIn’s professional focus, I believe this is a problem they can and should tackle.

Of course, Linkedin also needs to convince more of its users to join their LinkedIn accounts to their Twitter accounts–since that is their input source. But I suspect it’s mostly a matter of time and education–and hopefully the buzz around Signal will help raise awareness.

All in all, I see LinkedIn Signal as a great innovation and a big step forward for exploratory search and for Twitter. Congratulations to John Wang, Igor Perisic, and the rest of the LinkedIn search team on the launch!

To whom it may concern at the United States Patent Office:

Since completing my undergraduate studies in mathematics and computer science at the Massachusetts Institute of Technology (MIT) and my doctorate in computer science at Carnegie Mellon University (CMU), I have spent my entire professional life in software research and development. I have worked at large software companies, such as IBM, AT&T, and Google, and I also was a founding employee at Endeca, an enterprise software company where I served as Chief Scientist. I am a named inventor on eight United States patents, as well as on eighteen pending United States patent applications. I played an active role in drafting and prosecuting most of these patents. I have also been involved in defensive patent litigation, which in one case resulted in the re-examination of a patent and a final rejection of most of its claims.

As such, I believe my experience gives me a balanced perspective on the pros and cons of software patents.

As someone who has developed innovative technology, I appreciate the desire of innovators to reap the benefits of their investments. As a founding employee of  a venture-backed startup, I understand how venture capitalists and other investors value companies whose innovations are hard to copy. And I recognize how, in theory, software patents address both of these concerns.

But I have also seen how, in practice, software patents are at best a nuisance and innovation tax and at worst a threat to the survival of early-stage companies. In particular, I have witnessed the proliferation of software patents of dubious validity that has given rise to a “vulture capitalist” industry of non-practicing entities (NPEs), colloquially known as patent trolls, who aggressively enforce these patents in order to obtain extortionary settlements. Meanwhile, the software companies where I have worked follow a practice of accumulating patent portfolios primarily in order to use them as deterrents against infringement suits by companies that follow the same strategy.

My experience leads me to conclude that the only beneficiaries of the current regime are patent attorneys and NPEs. All other parties would be benefit if software were excluded from patent eligibility. In particular, I don’t believe that software patents achieve either of the two outcomes intended by the patent system: incenting inventors to disclose (i.e., teach) trade secrets, and encouraging investment in innovation.

First, let us consider the incentive to disclose trade secrets. In my experience, software patents fall into two categories. The first category focuses on interfaces or processes, avoiding narrowing the scope to any non-obvious system implementation details. Perhaps the most famous example of a patent in this category is Amazon’s “one-click” patent. The second category focuses on algorithm or infrastructure innovations that typically implemented as inside of proprietary closed-source software. An example in this category is the patent on latent semantic indexing, an algorithmic approach used in search and data mining applications. For the first category, patents are hardly necessary to incent disclosure, as the invention must be disclosed to realize its value. Disclosure is meaningful for patents in the second category, but in my experience most companies do not file such patents because they are difficult to enforce. Without access to a company’s proprietary source code, it is difficult to prove that said source code is infringing on a patent. For this reason, software companies typically focus on the first category of patents, rather than the second. And, as noted, this category of innovation requires no incentive for disclosure.

Second, let us ask whether software patents encourage investment in innovation. Specifically, do patents influence decisions by companies, individual entrepreneurs, or investors to invest time, effort, or money in innovation?

My experience suggests that they do not. Companies and entrepreneurs innovate in order to further their business goals and then file patents as an afterthought. Investors expect companies to file patents, but only because everyone else is doing it, and thus patents offer a limited deterrent value as cited above. In fact, venture capitalists investing in software companies are some of the strongest voices in favor of abolishing software patents. Here are some examples:

Chris Dixon, co-founder of software companies SiteAdvisor and Hunch and of seed-stage venture capital fund Founder Collective, says:

Perhaps patents are necessary in the pharmaceutical industry. I know very little about that industry but it would seem that some sort of temporary grants of monopoly are necessary to compel companies to spend billions of dollars of upfront R&D.

What I do know about is the software/internet/hardware industry. And I am absolutely sure that if we got rid of patents tomorrow innovation wouldn’t be reduced at all, and the only losers would be lawyers and patent trolls.

Ask any experienced software/internet/hardware entrepreneur if she wouldn’t have started her company if patent law didn’t exist. Ask any experienced venture investor if the non-existence of patent law would have changed their views on investments they made. The answer will invariably be no (unless their company was a patent troll or something related).

http://cdixon.org/2009/09/24/software-patents-should-be-abolished/

Brad Feld, co-founder of early-stage venture capital firms Foundry Group, Mobius Venture Capital and TechStars, says:

I personally think software patents are an abomination. My simple suggestion on the panel was to simply abolish them entirely. There was a lot of discussion around patent reform and whether we should consider having different patent rules for different industries. We all agreed this was impossible – it was already hard enough to manage a single standard in the US – even if we could get all the various lobbyists to shut up for a while and let the government figure out a set of rules. However, everyone agreed that the fundamental notion of a patent – that the invention needed to be novel and non-obvious – was at the root of the problem in software.

I’ve skimmed hundreds of software patents in the last decade (and have read a number of them in detail.) I’ve been involved in four patent lawsuits and a number of “threats” by other parties. I’ve had many patents granted to companies I’ve been an investor in. I’ve been involved in patent discussions in every M&A transaction I’ve ever been involved in. I’ve spent more time than I care to on conference calls with lawyers talking about patent issues. I’ve always wanted to take a shower after I finished thinking about, discussing, or deciding how to deal with something with regard to a software patent.

I’ll pause for a second, take a deep breath, and remind you that I’m only talking about software patents. I don’t feel qualified to talk about non-software patents. However, we you consider the thought that a patent has to be both novel AND non-obvious (e.g. “the claimed subject matter cannot be obvious to someone else skilled in the technical field of invention”), 99% of all software patents should be denied immediately. I’ve been in several situations where either I or my business partner at the time (Dave Jilk) had created prior art a decade earlier that – if the patent that I was defending against ever went anywhere – would have been used to invalidate the patent.

http://www.feld.com/wp/archives/2006/04/abolish-software-patents.html

Fred Wilson, managing partner of venture-capital firm Union Square Ventures:

Even the average reader of the Harvard Business Review has a gut appreciation for the fundamental unfairness of software patents. Software is not the same as a drug compound. It is not a variable speed windshield wiper. It does not cost millions of dollars to develop or require an expensive approval process to get into the market. When it is patented, the “invention” is abstracted in the hope of covering the largest possible swath of the market. When software patents are prosecuted, it is very often against young companies that independently invented their technology with no prior knowledge of the patent.

http://www.unionsquareventures.com/2010/02/software-patents-are-the-problem-not-the-answer.php

In summary, software patents act as an innovation tax rather than a catalyst for innovation. Perhaps it is possible to resolve the problems of software patents through aggressive reform. But it would be better to abolish software patents than to maintain the status quo.

Sincerely,

Daniel Tunkelang

For example, the principle motivation for the faceted search work I did at Endeca was to eliminate hurdles that discourage people from exploring information spaces. Most sites already offered user the ability to perform this exploration through advanced or parametric search interfaces–indeed, I recall some critics of faceted search objecting that it was nothing new. But there’s a reason that most of today’s consumer-facing sites place faceted search front and center while still relegating advanced search interfaces to an obscure page for power users. Faceted search offers users the fluidity and instant feedback that makes exploration natural for users. Once you’re used to it, it’s hard to live without it, whether your looking for real estate (compare Zillow.com to housing search on craigslist), library books (compare the Triangle Research Libraries Network to the Library of Congress), or art (compare to art.com to artnet).

Why is faceted search such a significant improvement over advanced or parametric search interfaces? Because it supports exploration at the speed of thought. If it takes you several seconds–rather than a single click–to refine a query, and if you have to repeatedly back off from pages with no results (aka dead ends), your motivation to explore a document collection fades quickly. But when that experience is fluid, you explore without even thinking about it. That is the promise (admittedly not always fulfilled) of faceted search.

Microsoft Live Labs director Gary Flake offered a similar message in his SIGIR 2010 keynote. He argued that we needed to replace our current discrete interactions with search engines into a mode of continuous, fluid interaction where the whole of data is greater than sum or parts. While he offered Microsoft’s Pivot client as an example of this vision, he could also have invoked the title of a book that Bill Gates wrote in 1999: Business @ the Speed of Thought. Indeed, anyone who has ever worked on data analysis understands that you ask fewer questions when you know you’ll have to wait for answers. Speed changes the way you interact with information.

And at Google, speed has been an obsession since day one. It makes the top 3 on the “Ten things we know to be true” list:

3. Fast is better than slow.

We know your time is valuable, so when you’re seeking an answer on the web you want it right away – and we aim to please. We may be the only people in the world who can say our goal is to have people leave our website as quickly as possible. By shaving excess bits and bytes from our pages and increasing the efficiency of our serving environment, we’ve broken our own speed records many times over, so that the average response time on a search result is a fraction of a second. We keep speed in mind with each new product we release, whether it’s a mobile application or Google Chrome, a browser designed to be fast enough for the modern web. And we continue to work on making it all go even faster.

People have made much of Google VP Marissa Mayer’s estimate that Google Instant will save 350 million hours of users’ time per year by shaving two to five seconds per search. That’s an impressive number, but I personally think it understates the impact of this interface change. Rather, I’m inclined to focus on a phrase I’ve seen repeatedly associated with Google Instant: “search at the speed of thought”.

What does that mean in practice? I see two major wins from Google Instant:

1) Typing speed and spelling accuracy don’t get in the way. For example, by the time you’ve typed [m n], you see results for M. Night Shyamalan, a name whose length and spelling might frustrate even his fans. A search for [marc z] offers results for Facebook CEO Mark Zuckerberg. Admittedly, the pre-Instant type-ahead suggestions already got us most of the way there, but the feedback of actual results offers not just guidace but certainty.

2) Users spend less–and hopefully no time–in a limbo where they don’t know if the system has understood the information-seeking intent they have expressed as a query. For example, if I’m interested in learning more about the Bob Dylan song “Forever Young“, I might enter [forever young] as a search query–indeed, the suggestion shows up as soon as I’ve typed in “fore”. But a glance at the first few instant results for [forever young] makes it clear that there are lots of songs by this title (including those by Rod Stewart and Alphaville–as well as the recent Jay Z song “Young Forever” that reworks the latter). Realizing that my query is ambiguous, I type the single letter “d” and instantly see results for the Dylan song. Yes, I could have backed out from an unsuccessful query and then tried again, but instant feedback means far less frustration.

Google Instant also makes it a little easier for users to explore the space of queries related to their information need, but exploration through instant suggestions is very limited compared to using related searches or the wonder wheel–let alone what we might be able to do with faceted web search. I’d love to see this sort of exploration become more fluid, but I recognize the imperative to maintain the simplicity of the search box. Good for us HCIR folks to know that there’s still lots of work to do on search interface innovation!

But, in short, speed matters. Instant communication has transformed the way we interact with one another–both personally and professionally. Instant search is more subtle, but I think it will transform the way we interact with information on the web. I am very proud of my colleagues’ collective effort to make it possible.

In 2007, I persuaded MIT graduate students Michael Bernstein and Robin Stewart (who was interning at Endeca that summer) to help organize the first Workshop on Human-Computer Information and Information Retrieval (HCIR 2007), which we held at MIT and Endeca. Its success convinced us to keep going, and we enjoyed record attendance at this year’s HCIR 2010, held at Rutgers University.

As the workshop has grown, we as organizers have realized that we need to invest a little in its online presence. A first step in that direction is a new site for the workshop: http://hcir.info/. The site contains all of the proceedings from the four annual workshops in one place. It is powered by Google Sites, which will make it easy for a bunch of us (and perhaps some of you) to collaboratively maintain it.

I hope everyone here finds the new site useful. Please feel free to come forward with ideas for improving it! But be warned–if you have a great idea, I might ask you to implement it yourself.

Image recognition is one of those problems that has presented long-standing challenges to computer scientists, despite being taken for granted by science fiction writers. Google Goggles represents one of the most audacious efforts to implement image recognition on on a massive scale.

Tonight, I had the pleasure of watching my colleague, David Petrou, present a live demo of Goggles to about 800 people who filled the NYU Skirball Center to attend the NY Tech Meetup. Many thanks to Nate Westheimer and Brandon Diamond for giving Google the opportunity to present this cool technology to a very engaged audience and in particular to show off some of the technology that Googlers are building here in New York City.

You can’t see the live demo in the slides, so I encourage you to view a recording of the presentation here.

Also, if you’re in the New York area and interested in hearing about upcoming Google NYC events, please sign up at http://bit.ly/googlenycevents.

One of the perks of blogging is that publishers sometimes send me review copies of new books. I couldn’t help but be curious about a book entitled “The Man Who Lied to His Laptop: What Machines Teach Us About Human Relationships“–especially when principal author Clifford Nass is the director of the Communications between Humans and Interactive Media (CHIMe) Lab at Stanford. He wrote the book with Corina Yen, the editor-in-chief of Ambidextrous, Stanford’s journal of design.

They start the book by reviewing evidence that people treat computers as social actors. Nass writes:

to make a discovery, I would find any conclusion by a social science researcher and change the sentence “People will do X when interacting with other people” to “People will do X when interacting with a computer”

They then apply this principle by using computers as confederates in social science experiments and generalizing conclusions about human-compter interaction to human-human interaction. It’s an interesting approach, and they present results about how people respond to praise and criticism, similar/opposite personalities, etc. You can get a taste of Nass’s writing from an article he published in the Wall Street Journal entitled “Sweet Talking Your Computer“.

The book is interesting and entertaining, and I won’t try to summarize all of its findings here. Rather, I’d like to explore its implications.

Applying the “computers are social actors” principle, they cite a variety of computer-aided experiments that explore people’s social behaviors. For example, they cite a Stanford study on how “Facial Similarity Between Voters and Candidates Causes Influence” , in which secretly morphing a photo of a candidate’s face to resemble the voter’s face induces a significantly positive effect on the voter’s preference. They also cite  another experiment on similarity attraction that varies a computer’s “personality” to be either similar or opposite to that of the experimental subject. A similar personality draws a more positive response than an opposite one, but the most positive response comes from the computer starts off with an opposite  personality and then adapts to conform to the personality of the subject. Imitation is flattery, and–as yet another of their studies shows–flattery works.

It’s hard for me to read results like these and not see creepy implications for personalized user interfaces. When I think about the upside of personalization, I envision a happy world where we see improvement in both effectiveness and user satisfaction. But clearly there’s a dark side where personalization takes advantage of knowledge about users to manipulate their emotional response. While such manipulation may not be in the users’ best interests, it may leave them feeling more satisfied. Where do we draw the line between user satisfaction and manipulation?

I’m not aware of anyone using personalization this way, but I think it’s a matter of time before we see people try. It’s not hard to learn about users’ personalities (especially when so many like taking quizzes!), and apparently it’s easy to vary the personality traits that machines project in generated text, audio, and video. How long will it before people put these together? Perhaps we are already there.

O brave new world that has such people and machines in it. Shakespeare had no idea.

With 70 registered attendees, it was the biggest HCIR workshop we have held. Rutgers was a gracious host, providing space not only for the all-day workshop but also for a welcome reception the night before.

And, based on an informal survey of participants, I can say with some semblance of objectivity that this was the best HCIR workshop to date.

The opening “poster boaster” session was particularly energetic. There was no award for best boaster, but Cathal Hoare won an ovation by delivering his boaster as a poem:

If a picture is worth a thousand words

Surely to query formulation a photo affords

The ability to ask ‘what is that’ in ways that are many

But for years we have asked how can-we

Narrow the search space so that in reasonable time

We can use images to answer questions that are yours and mine

In my humble poster I will describe

How recent technology and users prescribe

A solution that allows me to point and click

And get answers so that I don’t feel so thick

About my location and my environment

And to my touristic explorations bring some enjoyment

Now if after all that you feel rather dazed

Please come by my poster and see if you are amazed….

As in past years, we enlisted a rock-star keynote speaker–this time, Google UX researcher Dan Russell. His slides hardly do justice to his talk–especially without the audio and video–but I’ve embedded them here so that you can get a flavor for his presentation on how we need to do more to improve the searcher.

We accepted six papers for the presentation sessions–sadly, one of the presenters could not make it because of visa issues. The five presentations covered a variety of topics relating to tools, models, and evaluation for HCIR. The most intriguing of these (to me, at least) was a presentation by Max Wilson about “casual-leisure searching”–which he argues breaks our current models of exploratory search. Check out the slides below, as well as Erica Naone’s article in Technology Review on “Searching for Fun“.

I’ve been exchanging emails with Dhiti co-founder Bharath Mohan about Nuggetize, an intriguing interface that surfaces “nuggets” from a site to reduce the user’s cost of exploring a document collection. Specifically Nuggetize targets research scenarios where users are likely to assemble a substantial reading list before diving into it. You can try Nuggetize on the general web or on a particular site that has been “nuggetized”, e.g., a blog like this one or Chris Dixon’s.

I’m always happy to see people building systems that explicitly support exploratory search (and am looking forward to seeing the HCIR Challenge entries in a week!). Regular readers may recall my coverage of Cuil, Kosmix, and Duck Duck Go. And of course I helped build a few of my own at Endeca. So what’s special about Nuggetize?

Mohan describes it as a faceted search interface for the web. I’ll quibble here–the interface offers grouped refinement options, but the groups don’t really strike me as facets. Moreover, the interface isn’t really designed to explore intersections of the refinement options–rather, at any given time, you see the intersection of the initial search and a currently selected refinement. But it is certainly an interface that supports query refinement and exploration.

The more interesting features are the nuggets and the support for relevance feedback.

The nuggets are full sentences, and thus feel quite different from conventional search-engine snippets. Conventional snippets serve primarily to provide information scent, helping users quickly determine the utility of a search result without the cost of clicking through to it and reading it. In contrast the nuggets are document fragments that are sufficiently self-contained to communicate a coherent thought. The experience suggests passage retrieval rather than document retrieval.

The relevance feedback is explicit: users can thumbs-up or thumbs-down results. After supplying feedback, users can refresh their results (which re-ranks them) and are also presented with suggested categories to use for feedback (both positive and negative). Unfortunately, the research on relevance feedback tells us that, helpful as it could be to improving user experience, users don’t bite. But perhaps users in research scenarios will give it a chance–especially with the added expressiveness and transparency of combining document and category feedback.

Overall it is a slick interface, and it’s nice seeing the various ideas Mohan and his colleagues put together. There’s certainly room for improvement–particularly in the quality of the categories, which sometimes feel like victims of polysemy. Open-domain information extraction is hard! Some would even call it a grand challenge.

Mohan reads this blog (he reached out to me a few months ago via a comment), and I’m sure he’d be happy to answer questions here.

If you’re a search engine junkie like me, you’ve probably heard about Blekko, a search engine that has been percolating for over two years and recently launched a private beta. If not, I encourage you to watch the TechCrunch video I’ve embedded above. You can join the beta by following them on Twitter. I did that earlier this week, and my invitation arrived via a direct message the next day.

Blekko’s main differentiating feature is that it supports “slashtags”. These aren’t the same as the Twitter microsyntax proposed by Chris Messina and named by Chris Blow. Rather, they are a way for users to “spin” their search results using a variety of filters. For example, [climate /liberal] and [climate /conservative] return very different results, because they are restricted to different sets of sites.

In addition to providing a set of curated slashtags, Blekko allows users to define their own slashtags by specifying the sets of sites to be included. There’s a social aspect here too: you can use (and follow) other users’ slashtags. Blekko also has some special slashtags that don’t act as site filters, e.g., /date shows recent results and /seo offers indexing information about web sites.

Blekko emphasizes two characteristics that I find very appealing: transparency and user control. While they do not disclose their relevance ranking algorithm, they do expose some of the information they use to compute it. More significantly, their emphasis on slashtags de-emphasizes default ranking, but rather encourages users to take more responsibility in the information seeking process. Very HCIR!

I like the concept. But I’m not sure how I feel about the execution. I have three main concerns.

First, the set of slashtags is somewhat haphazard–to be expected in a beta, but I’m not sure how it will evolve. I’d love to see a vocabulary collectively (and transparently) curated like Wikipedia, but I fear it will look more like social tagging site Delicious, which is a case study in the “vocabulary problem“. As any information scientist can tell you, managing vocabularies is hard!

Second, I’m not sure if site filters are the right model. What happens to sites with heterogeneous content? Or to sites that have one-hit wonders and therefore are unlikely to show up in any slashtags? I’d prefer to see the sites used as seeds to train classifiers that could then be applied to the entire index. Something a bit more like what Miles Efron implemented in this research–only on a much larger scale and applied at a page rather than site level.

Third, I think there’s a third ingredient that is essential to complement transparency and user control: guidance. As a user, I need to know what slashtags would lead me to interesting results, and ideally I’d want some kind of preview to make exploration as low-cost as possible.

I know I’m asking for a lot–especially from an ambitious startup that has just launched its private beta. But I think the stakes are high in this space, and going easy on a newcomer is no favor. I offer the tough love of a critic who would really like to see this kind of vision succeed.

If you are are interested in attending and haven’t already registered, please let me know as soon as possible via email or Twitter (speaking of which, follow the #hcir2010 hash tag). We’re making the remaining slots available to the community on a first-come, first-serve basis.

Google user experience researcher Dan Russell will be delivering this year’s keynote on “Why is search sometimes easy and sometimes hard? Understanding serendipity and expertise in the mind of the searcher“.

Here is the list of accepted papers:

Oral Presentations

• VISTO: for Web Information Gathering and Organization
  Anwar Alhenshiri, Carolyn Watters, and Michael Shepherd (Dalhousie University)
• Time-based Exploration of News Archives
  Omar Alonso (Microsoft Corporation), Klaus Berberich (Max-Planck Institute for Informatics), Srikanta Bedathur (Max-Planck Institute for Informatics), and Gerhard Weikum (Max-Planck Institute for Informatics)
• Combining Computational Analyses and Interactive Visualization to Enhance Information Retrieval
  Carsten Goerg, Jaeyeon Kihm, Jaegul Choo, Zhicheng Liu, Sivasailam Muthiah, Haesun Park, and John Stasko (Georgia Institute of Technology)
• Impact of Retrieval Precision on Perceived Difficulty and Other User Measures
  Mark Smucker and Chandra Prakash Jethani (University of Waterloo)
• Exploratory Searching As Conceptual Exploration
  Pertti Vakkari (University of Tampere)
• Casual-leisure Searching: The Exploratory Search Scenarios that Break our Current Models
  Max L. Wilson (Swansea University) and David Elsweiler (University of Erlangen)

HCIR Challenge Reports

• Search for Journalists: New York Times Challenge Report
  Corrado Boscarino, Arjen P. de Vries, and Wouter Alink (Centrum Wiskunde and Informatica)
• Exploring the New York Times Corpus with NewsClub
  Christian Kohlschütter (Leibniz Universität Hannover)
• Searching Through Time in the New York Times
  Michael Matthews, Pancho Tolchinsky, Roi Blanco, Jordi Atserias, Peter Mika, and Hugo Zaragoza (Yahoo! Labs)
• News Sync: Three Reasons to Visualize News Better
  V.G. Vinod Vydiswaran (University of Illinois), Jeroen van den Eijkhof (University of Washington), Raman Chandrasekar (Microsoft Research), Ann Paradiso (Microsoft Research), and Jim St. George (Microsoft Research)
• Custom Dimensions for Text Corpus Navigation
  Vladimir Zelevinsky (Endeca Technologies)
• A Retrieval System Based on Sentiment Analysis
  Wei Zheng and Hui Fang (University of Delaware)

Research Posters

• Improving Web Search for Information Gathering: Visualization in Effect
  Anwar Alhenshiri, Carolyn Watters, and Michael Shepherd (Dalhousie University)
• User-oriented and Eye-Tracking-based Evaluation of an Interactive Search System
  Thomas Beckers and Norbert Fuhr (University of Duisberg-Essen)
• Exploring Combinations of Sources for Interaction Features for Document Re-ranking
  Emanuele Di Buccio (University of Padua), Massimo Melucci (University of Padua), and Dawei Song (The Robert Gordon University)
• Extracting Expertise to Facilitate Exploratory Search and Information Discovery: Combining Information Retrieval Techniques with a Computational Cognitive Model
  Wai-Tat Fu and Wei Dong (University of Illinois at Urbana-Champaign)
• An Architecture for Real-time Textual Query Term Extraction from Images
  Cathal Hoare and Humphrey Sorensen (University College Cork)
• Transaction Log Analysis of User Actions in a Faceted Library Catalog Interface
  Bill Kules (The Catholic University of America), Robert Capra (University of North Carolina at Chapel Hill), and Joseph Ryan (North Carolina State University Libraries)
• Context in Health Information Retrieval: What and Where
  Carla Lopes and Cristina Ribeiro (University of Porto)
• Tactics for Information Search in a Public and an Academic Library Catalog with Faceted Interfaces
  Xi Niu and Bradley M. Hemminger (University of North Carolina at Chapel Hill)

Position Papers

• Understanding Information Seeking in the Patent Domain and its Impact on the Interface Design of IR Systems
  Daniela Becks, Matthias Görtz, and Christa Womser-Hacker (University of Hildesheim)
• Better Search Applications Through Domain Specific Context Descriptions
  Corrado Boscarino, Arjen P. de Vries, and Jacco van Ossenbruggen (Centrum Wiskunde and Informatica)
• Layered, Adaptive Results: Interaction Concepts for Large, Heterogeneous Data Sets
  Duane Degler (Design for Context)
• Revisiting Exploratory Search from the HCI Perspective
  Abdigani Diriye (University College London), Max L. Wilson (Swansea University), Ann Blandford (University College London), and Anastasios Tombros (Queen Mary University London)
• Supporting Task with Information Appliances: Taxonomy of Needs
  Sarah Gilbert, Lori McCay-Peet, and Elaine Toms (Dalhousie University)
• A Proposal for Measuring and Implementing Group’s Affective Relevance in Collaborative Information Seeking
  Roberto González-Ibáñez and Chirag Shah (Rutgers University)
• Evaluation of Music Information Retrieval: Towards a User-Centered Approach
  Xiao Hu (University of Illinois at Urbana Champaign) and Jingjing Liu (Rutgers University)
• Information Derivatives: A New Way to Examine Information Propagation
  Chirag Shah (Rutgers University)
• Implicit Factors in Networked Information Feeds
  Fred Stutzman (University of North Carolina at Chapel Hill)
• Improving the Online News Experience
  V. G. Vinod Vydiswaran (University of Illinois) and Raman Chandrasekar (Microsoft Research)
• Breaking Down the Assumptions of Faceted Search
  Vladimir Zelevinsky (Endeca Technologies)
• A Survey of User Interfaces in Content-based Image Search Engines on the Web
  Danyang Zhang (The City University of New York)

You can also download the full proceedings here.

So there’s no question that community question answering is hot. The question is why? In particular, is community question answering a step forward or backward relative to today’s search engines, or is it something different?

Regarding Facebook Questions, Jason Kincaid writes in TechCrunch:

Given its size, it won’t take long for Facebook to build up a massive amount of data — if that data is consistently reliable, Questions could turn into a viable alternative to Google for many queries.

That’s a big if.  But I think the bigger caveat is the vague quantifier “many”. The success of community question answering services will depend on how these services position themselves relative to users’ information needs. Anyone arguing that these services can or should replace today’s web search engines might want to consider the following examples of information needs that are typical of current search engine use:

• How do I get an iPhone case?
• Who sings the “choco latte” song?
• What movies are playing in my neighborhood?
• How do I get to Boston from New York?
• What is the best selling netbook?
• Who offers the best cell phone reception in New York?
• What was the score in the North Korea – Portugal game?

I hope I don’t have to keep going to convince you that web search engines have earned their popularity by serving a broad class of information needs (i.e., answer lots of questions)–and that’s without even using the wide variety of personalized and social features that web search engines are rapidly developing.

The common thread in the above questions is that they focus on objective information. In general, such questions are effectively and efficiently answered by search engines based on indexed, published content (including “deep web” content made available to search engines via APIs). There’s a lot of work we can do to improve search engines, particularly in the area of supporting query formulation. But it seems silly and wasteful to route such questions to other people–human beings should not be reduced to performing tasks at which machines excel.

That said, I agree with Kincaid that there are many information needs that are well addressed by  community question answering. In particular:

• Questions for which point of view is a feature, not a bug. Review sites succeed when they provide sincere, informed personal reactions to products and services. Similarly, routing questions to people makes sense either when we care about the answerer’s a point of view. For some questions, I want the opinion of someone who shares my taste (which is what Hunch is pursuing with its “taste graph“). For others, I want a diversity of expert opinions–for which I might turn to Aardvark (which tries to route questions to topic experts), Quora (where people follow particular topics), or LinkedIn Answers. Over time, the answers to many such questions can be published and indexed–and indeed some answers sites receive a large share of their traffic from search engines.
• Niche topics. As much as web search as improved information accessibility for the “long tail” of published information, the effectiveness of web search can be highly variable for the most obscure information needs. Moreover, this effectiveness depends significantly on the user: some people are better at searching than others, especially in their areas of domain expertise. Social search can help level the playing field. Much as Wikipedia has surfaced much of the expertise at the head of the information distribution, community question answering can help out in the tail.
• Community for its own sake. Even in cases where search engines are more effective and efficient than community question answering services, some people prefer to participate in a social exchange rather than to conduct a transaction with an impersonal algorithm. Indeed, researchers at Aardvark found that many of the questions posed through their service (pre-acquisition) could be answered successfully using Google. I’ll go out on a limb and assume that Aardvark’s users were early technology adopters who are quite conversant with search engines–but in some case chose to use a social alternative simply because they wanted to be social.

Conclusions? Community question answering may be overhyped right now, but it isn’t a fad. There are broad classes of subjective information needs that require a point of view, if not a diversity of views. And even if much of the use of community question answering sites is mediated by search engines indexing their archives, there will always be a need for fresh content. I also believe that social search will continue to be valuable for niche topics, since neither search engines nor searchers will ever be perfect.

But I think the biggest open question is whether people will favor community question answering simply to be social. I conjecture that, by very publicly integrating community question answering into is social networking platform, Facebook is testing the hypothesis that it can turn information seeking from a utilitarian individual task into an entertaining social destination. Given Facebook’s highly engaged user population, we won’t have to wait long to find out.

• The new frontiers of Web search: going beyond the 10 blue links
  Ricardo Baeza-Yates, Andrei Broder, Yoelle Maarek, and Prabhakar Raghavan, Yahoo! Labs
  
• Cross-Language Information Retrieval in the Legal Domain
  Samir Abdou and Thomas Arni, Eurospider
• Building and Configuring a Real-Time Indexing System
  Garret Swart, Ravi Palakodety, Mohammad Faisal, Wesley Lin, Oracle
  
• Lessons and Challenges from Product Search
  Daniel E. Rose, A9.com (Amazon)
  
• Being Social: Research in Context-aware and Personalized Information Access @ Telefonica
  Xavier Amatriain, Karen Church and Josep M. Pujol, Telefónica
  
• Searching and Finding in a Long Tail Marketplace
  Neel Sundaresan, eBay
  
• When No Clicks are Good News
  Carlos Castillo, Aris Gionis, Ronny Lempel, and Yoelle Maarek, Yahoo! Research

I missed the Eurospider and Oracle talks, but otherwise I spent the afternoon enjoying these sessions. The slides, along with all of the keynote slides, are available here.

Some highlights from the talks I attended:

• Andrei Broder, a pioneer of Web IR and author of the highly cited “Taxonomy of Web Search“,  enumerated a half-dozen challenges for web search to move from its current state to one that not only accomplishes semantic analysis but also supports task completion. Naturally, the one that appeals to me is the need for search engines to move beyond query suggestion and truly engage the user in a dialog.
• Dan Rose talked about the challenges of product search, and in particular the blessing and curse of implementing search applications for structured data (something that I’m very familiar with from my previous role at Endeca). He also warned of the dangers of over-interpreting behavioral data, e.g., a site change that increases revenue does not necessarily imply a better user experience (it could just be favoring higher-priced inventory), and may ultimately alienate customers.
• Xavier Amatriain focused on social search, and talked about how, as we’ve turned to context to help mitigate information overload, we find ourselves confronted with the new problem of context overload. Specifically, he cited the research questioning the wisdom of the crowd, and proposed the wisdom of the (expert) few as a better alternative.
• Neel Sundaresan offered an interesting tour of eBay Research Labs prototypes, including the BayEstimate that helps sellers improve listing titles by discovering the keywords are both representative of the item and used in buyers’ queries.
• Finally, Carlos Castillo offered a nice approach to discover when search engine abandonment is “good abandonment“: identify a subset of “tenacious” users who almost never abandon searches and measure their abandonment–since it is almost certain to be the good kind.

All in all, I was very impressed with the quality of the Industry Track, and gratified to see how it had improved on the program I put together last year. Given the key role that industry plays in information retrieval, I think it is important that the top-tier IR conference promote the best that industry has to offer.

But this year the SIGIR 2010 Industry Track broke new ground. The keynotes were from some of the most senior technologists at the world’s largest web search engines:

• William Chang, Chief Scientist at Baidu
• Yossi Matias, Head of Google’s Israel R&D Center
• Jan Pedersen, Chief Scientist for Core Search at Microsoft (Bing)
• Ilya Segalovich, CTO and Co-Founder of Yandex

I won’t attempt to provide much detail about these presentations, first because I’m hoping they will all be they have all been posted online and second because Jeff Dalton has already done an excellent job of posting live-blogged notes. Rather, I’ll offer a few reactions.

William’s presentation on the “Future Search: From Information Retrieval to Information Enabled Commerce” unsurprisingly focused on the Chinese search-related market. While the topic of  Google in China was an elephant in the room, it did not surface even obliquely in the presentation–and I commend William for taking the high road. As for Baidu itself, its most interesting innovation is Aladdin, an open search platform that allows participating webmasters to submit query-content pairs.

Yossi’s presentation on “Search Flavours at Google” was a tour de force of Google’s recent innovations in the search and data mining space. The search examples most focused on the challenges of incorporating context into query understanding–where context might involve geography, time, social network, etc. But some of the more impressive examples showed off using the power of data to predict the present. More than anything, his presentation made clear that Google is doing a lot more than returning the traditional ten blue links.

Jan talked about “Query Understanding at Bing”. I really hope he makes these slides available, since they do a really nice job of describing a machine learning based architecture for processing search queries. To get an idea of this topic, check out Nick Craswell’s presentation from last year’s SIGIR.

Finally Ilya talked about “Machine Learning in Search Quality at Yandex”, the largest search engine in Russia. He described the main challenge in Russia as handling the local aspects of search: he gave as an example that, if you’re in a small town in Russia, then local results in Moscow may as well be on the moon. Local search is a topic close to my heart, not least of which because it is my day job! Ilya’s talked focused largely on Yandex’s MatrixNet implementation of learning to rank. What I’m surprised he didn’t mention is the challenges of data acquisition–in general, for domains beyond the web, obtaining high-quality data is often a much bigger challenge than filtering and ranking it.

All in all, the four keynotes collectively offered an excellent state-of-the-search-engine address.  As with last year, the industry track talks were the most popular morning sessions, and the speakers delivered the goods.

In the morning session, I attended three talks. The first, “Geometric Representations for Multiple Documents” by Jangwon Seo and Bruce Croft, looks at the problem of representing combinations of documents in a query model. It provides both theoretical and experimental evidence that geometric means work better than arithmetic means for representing such combinations. The second, “Using Statistical Decision Theory and Relevance Models for Query-Performance Prediction” by Anna Shtok, Oren Kurland, and David Carmel, shows the efficacy of a utility estimation framework comprised of relevance models, measures like query clarity to estimate the representativeness of relevance models, and similarity measures to estimate the similarity or correlation between two ranked lists. The authors demonstrated significant improvements from the framework over simply using the representativeness measures for performance prediction. The third paper, “Evaluating Verbose Query Processing Techniques” by Samuel Huston and Bruce Croft, showed that removing “stop structures”, a generalization of stop words, could significantly improve performance on long queries. Interestingly, the authors evaluated their approach on “black box” commercial search engines Yahoo and Bing without knowledge of their retrieval models.

In the session after lunch, I mostly attended talks from the session on user feedback and user models. The first, “Incorporating Post-Click Behaviors Into a Click Model” by Feimin Zhong, Dong Wang, Gang Wang, Weizhu Chen, Yuchen Zhang, Zheng Chen, and Haixun Wang, proposed and experimentally validated a click model to infer document relevance from post-click behavior like dwell time that can be derived from logs. The second, “Interactive Retrieval Based on Faceted Feedback” by Lanbo Zhang and Yi Zhang, described an approach using facet values for relevance and pseudo-relevance feedback. It’s interesting work, but I think the authors should look at work my colleagues and I presented at HCIR 2008 on distinguishing whether facet values are useful for summarization or for refinement. The third, “Understanding Web Browsing Behaviors through Weibull Analysis of Dwell Time” by Chao Liu, Ryen White, and Susan Dumais, offered an elegant model of dwell time and used it to predict dwell time distribution from page-level features. Finally, I attended one talk from the session on retrieval models and ranking: “Finding Support Sentences for Entities” by Roi Blanco and Hugo Zaragoza. They present a novel approach of generalizing snippets to interfaces that offer named entities (e.g., people) as supplements to the search results. I am excited to see research that could make richer interfaces more explainable to users.

I spend the last session of the day listening to a couple of talks about users and interactive IR. The first was “Studying Trailfinding Algorithms for Enhanced Web Search” by Adish Singla, Ryen White, and Jeff Huang, turned out to be the best-paper winner. This work extends previous work that Ryen and colleagues have done on search trails and showed results of various trailfinding algorithms that outperform the trails users follow on their own. The second, “Context-Aware Ranking in Web Search” by Biao Xiang, Daxin Jiang, Jian Pei, Xiaohui Sun, Enhong Chen, and Hang Li, analyzes requerying behavior as reformulation, specialization, generalization, or general association, and demonstrates that knowing or inferring which the user is doing significantly improves ranking of the second query’s results.

The day wrapped up with a luxurious banquet at the Hotel Intercontinental, near the Nations Plaza. After sweating through conference sessions without air conditioning, it was a welcome surprise to enjoy great food in such an elegant setting.