Author: Daniel Tunkelang

Earlier this week, I had the good fortune to attend the New York Semantic Web Meetup, which featured three excellent presentations. I’ll confess that I primarily attended the event in order to learn more about open data platform Factual, particularly to see how it compares to Freebase and Google Squared.

But I found the other two presentations, “The Evolution of Social Search” by Nitya Narasimhan and “User Interfaces for the Semantic Web” by Duane Degler, even more compelling. Hopefully I’m not just biased because they both mentioned me in their slides!

I really wish Nitya’s session had been recorded as a video, but the slides embedded above will have to suffice for those who couldn’t see it live. Hopefully they communicate Nitya’s framing of the social search space. She does a great job of weaving together the various strands of social search: people as sensors promoting “real-time” content, social filters to reflect personalized notions of trust, and routers to leverage the collective intelligence of crowds. She gives tons of examples and links for further reading. Hopefully I’ll get her to give this talk again–with less stringent time constraints–and record it for online viewing.

I’ve been following Hunch for a while, and my impression has evolved from the initial skepticism with which I greeted it a year ago (to the day!). Given the track records of co-founders Caterina Fake and Chris Dixon, perhaps I should have expected their success at obtaining traffic and funding.  But what interests me more is that they are doing interesting things with data mining and putting a new twist on social media analytics.

For those unfamiliar with Hunch, it is a decision engine (cf. [real decision engine] vs. [decision engine]). For example, it can help you decide whether to buy an iPad or how to name your baby. While it’s not clear to me how much people are using Hunch for utility vs. entertainment, Hunch is certainly accumulating users–as well as the data that those users volunteer.

Hunch recently released two applications that mash up that data with the Twitter follower graph. The first is a “Twitter Predictor Game” that attempts to calculate your taste profile from your Twitter id and then predict how you’ll answer Hunch’s taste questions. Just to keep the game honest, you can look at the Hunch’s guess either before or after you provide your answer. The second is called “Twitter Follower Stats“: given a Twitter user, it reports the salient information it has inferred about that user’s followers (e.g., @maddow vs. @karlrove).

I think this stuff is neat, and a great testament to the “unreasonable effectiveness of data“. The question-answer data still feels a bit sparse for my taste, and I suspect there’s still room for more dimensionality reduction. I’m sure Hunch CTO Matt Gattis and colleagues are working on it! Also, it would be neat to direct the follower analytics rather than simply see the ones that Hunch deems most salient.

In summary, Hunch is keeping it interesting. Definitely a startup to watch and learn from.

http://blip.tv/play/AYHOqGwC

Mathew Ingram posted an interview with Craig Newmark (the Craig of craigslist fame) in which the latter argued that what the web needs is a “distributed trust network” to manage our online reputations. As it happens, this is an idea that has occupied me for several years. So I figured it was about time that I shared my thoughts on the subject.

When we think of how trust works online, two of the most prominent examples are Google’s PageRank measure and eBay’s feedback scores. But neither of these measures addresses what I think Craig has in mind. PageRank is a great way of using citation analysis to determine the most authoritative citations, but the trust in a page should consider its out-links (i.e., can we trust the page not to point us to untrustworthy ones?) and not just its in-links. eBay’s feedback scores have a different problem: they count positive and negative ratings without considering the social network of buyers and sellers–and approach that is vulnerable to fraud through shill ratings. Incidentally,LinkedIn recommendations have a similar weakness if viewed in strictly quantitative terms, but the potential for abuse is mitigated by the endorsements being signed–and by their being more than just binary or numerical ratings. Incidentally, here’s a site you can use if you’re too lazy to actually write the recommendations yourself.

But I digress. Propagation of trust does seem like the perfect application to build on top of social networks. Consider any problem that involves getting advice to inform a decision. If we regularly solicit advice from our first-degree connections, then we should be able to learn over time whose advice we can trust. We can then vouch for these connections, which offers the connections who trust us a basis for trusting their second-degree connections through us. And so forth through our social network. Of course, trust is not irrevocable: loss of trust should propagate similarly.

I’ve talked about this problem with two of the leading experts on social networks, Jon Kleinberg and Prabhakar Raghavan, and as far as I know no one has built a system along these principles. In economic terms, I envision a system where a person’s reputation truly is his or her coin. One person might think of bribing one another to exploit the latter’s established reputation, but a rational person with a strong reputation would demand an exorbitant bribe to put that reputation at risk.

Of course, a lot of information would have to propagate throughout the social network–and be stored–for this system to work. Regardless of how the information is abstracted, such a reputation index would raise thorny privacy issues. Nonetheless, I don’t know if we can build a reputation system that is entirely privacy-preserving–since reputation is an inherently public mechanism. In addition, any such system would have to consider the implications of defamation laws. These are some major hurdles!

Nonetheless, I agree wholeheartedly with Craig that a distributed trust network could be “the killingest of killer apps”. I just hope we can find a way to build and use it!

Note: Chris Rines suggested I look at Advogato’s Trust Metric, and a quick investigation led me to the Wikipedia entry for trust metric. Looks like I have some homework to do!

In a post on ReadWriteWeb, Sarah Perez summarizes “Measuring User Influence in Twitter: The Million Follower Fallacy“, a recent research paper by Meeyoung Cha, Hamed Haddadi, Fabricio Benevenuto, and Krishna Gummadi. The punch line should hardly be surprising to regular readers here given my variety of rants on the subject: follower count isn’t great measure of influence.

The authors focus on measuring three quantities: followers (which they call indegree), retweets, and mentions. Their main results is that, while the number of followers is strongly correlated to the numbers of retweets and mentions for the general user population, the correlation is much weaker for the users with high follower counts, e.g., in the top 10%. Indeed, the authors believe that the correlation for the general population is “an artifact of the tied ranks among the least influential users, e.g., many of the least connected users also received zero retweet and mention.”

The authors further note that:

Across all three measures, the top influentials were generally recognizable public figures and websites. Interestingly, we saw marginal overlap in these three top lists. These top-20 lists only had 2 users in common: Ashton Kutcher and Puff Daddy. The top-100 lists also showed marginal overlap, as shown in Figure 1, indicating that the three measures capture different types of influence.

The authors ultimately conclude that:

• Follower count represents a user’s popularity, but is not related to notions of influence such as engaging audience, i.e., retweets and mentions.
• Retweets are driven by the content value of a tweet, favoring mainstream news organizations.
• Mentions are driven by the name value of the user, favoring celebrities.

I can’t argue with any of the above, but I do wonder if any of them are ideal measures of influence. All three measures are easy to game–and none of them model the scarcity of user attention, which is the motivating principle of TunkRank. Nor do they ground “influence” in any outcome external to Twitter.

Still, it’s an interesting negative result. If nothing else, it helps reinforce the argument that follower count isn’t a useful measure–at least once you get beyond the very low end of the range.

Let me preface this post with a clear disclaimer: I work at Google, but the views I express on this blog are my own personal views.

Last week, Google head of webspam Matt Cutts posted a full-throated defense of Google’s transparency on Google’s European Policy Blog in response to complaints that a few companies raised to the European Commission. Long-time readers of my blog know that I’m a big fan of search engine transparency and have made my own calls on this blog for Google to be more transparent. The fact that I work at Google now doesn’t change my values. But being on the inside has informed my perspective.

In particular, as Matt elaborates in his post, Google deserves more credit for transparency than it often gets from its critics. For example, Google has published:

• “The Anatomy of a Large-Scale Hypertextual Web Search Engine“, which not only details the formula for PageRank, but also mentions other signals that Google uses to rank search results: anchor text, location of query terms within documents, proximity of query terms, etc.
• details of its key infrastructure innovations: MapReduce, the Google File System, Bigtable, and Protocol Buffers
• hundreds of research papers by Googlers in diverse areas of computer science

He goes onto describe the various webmaster tools and social media resources that Google has made available. The popularity of these tools is a testament to their utility.

Still, as Matt points out:

we don’t think it’s unreasonable for any business to have some trade secrets, not least because we don’t want to help spammers and crackers game our system. If people who are trying to game search rankings knew every single detail about how we rank sites, it would be easier for them to ‘spam’ our results with pages that are not relevant and are frustrating to users — including porn and malware sites.

As I blogged back in 2008, I still hope that someday we won’t need to have to rely on a relevance analog of security through obscurity in order to deter spam and abusive SEO practices. But I recognize that we haven’t developed such an analog, and hence that complete transparency today for web search ranking algorithms would have a far greater downside than upside for ordinary users.

I suspect that a prerequisite for complete transparency in search requires moving from a ranking-based retrieval approach to a set-based approach. For many web search information needs (e.g., navigational queries), it’s hard to see how users would benefit from such a radical change. For queries that represent more exploratory information needs, a set-based approach would be (at least in my view) far preferable to one based on ranking. But there’s a lot of work to do on the content side before such exploratory interfaces for the web are usable.

In summary, I’m happy to see Matt taking a public stand in Google’s defense. I don’t always agree with my employer’s decisions, but I do believe that my colleagues act in good faith and with good intentions. I understand how many people–especially site owners–fixate on whatever Google keeps secret. In a world where so many people compete for attention, information is power. Google tries to provide maximum quality to users while keeping the playing field level for site owners. As Google Fellow Amit Singhal points out, “this stuff is tough“.

A topic with which I developed an obsession in my last few years at Endeca is understanding how to predict query difficulty and performance–performance in the information retrieval sense meaning results quality, not computational efficiency. If only we knew how well a search engine would do–or did–in meeting the user’s information need, we might adapt the user experience to reflect our degree of confidence.

I was particularly interested in work related to the query clarity score initially proposed by Steve Cronen-Townsend, Yun Zhou, and Bruce Croft in a 2002 paper entitled “Predicting Query Performance“. But there is a wide variety of work in this area, including methods to predict performance either before or after results retrieval.

Happily, Claudia Hauff just published a dissertation on this topic, entitled “Predicting the Effectiveness of Queries and Retrieval Systems“. It is very well written, and I recommend it to anyone interested in learning more about this subject. She presents not only her own original research, but also a comprehensive analysis of others’ efforts.

Here is an excerpt from the abstract:

In this thesis we consider users’ attempts to express their information needs through queries, or search requests and try to predict whether those requests will be of high or low quality. Intuitively, a query’s quality is determined by the outcome of the query, that is, whether the retrieved search results meet the user’s expectations. The second type of prediction methods under investigation are those which attempt to predict the quality of search systems themselves. Given a number of search systems to consider, these methods estimate how well or how poorly the systems will perform in comparison to each other.

I look forward to seeing researchers continue to build on these results, and I am excited for the day when search engines are more reflective on their own strengths and weakness.

We’re gearing up to officially announce the HCIR 2010 workshop, but I wanted to give folks here a heads up, as well as to put out a call for a volunteer.

The Fourth Workshop on Human-Computer Interaction and Information Retrieval will take place on August 22nd at Rutgers University in New Brunswick, NJ. It will be an independent workshop as in previous years, but this year we are co-locating it with the Third Information Interaction in Context conference (IIiX 2010).

We’ve already lined up Google’s Dan Russell as a keynote speaker and are close to circulating a call for participation. We’re also planning to introduce something new to the workshop this year: an HCIR challenge! Participants will build applications around a specific data set that demonstrate the use of HCIR techniques. We’ll announce the data set and the challenge details as soon as we’ve confirmed the licensing details.

Meanwhile, we’re looking for a volunteer to help us build a baseline index for the challenge data set. Participants will be allowed–but not required–to use this index as a starting point for their entries. The volunteer should be comfortable using open-source packages Lucene or Solr. If you are interested in being that volunteer, please let me know, and I’ll be happy to share more details.

in an information-rich world, the wealth of information means a dearth of something else: a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it (Simon, 1971)

I hope everyone agrees that attention is a scarce good. But I’m curious how people measure it. After all, if we’re going to talk about an economic good being scarce, we ought to quantify it!

One approach is to measure attention at a specific moment in time, measuring how much of our instantaneous cognitive capacity we devote to a task. This approach is useful for evaluating a user interface–in particular, for determining how users allocate their attention among the various interface elements. Another approach is to measure attention in units of time, e.g., how many of our waking hours do we devote to a particular activity. This latter strikes me as more of what Herb Simon had in mind.

We can interpret the two definitions as equivalent–after all, cumulative attention devoted to a task is simply the sum (or integral) of instantaneous attention over time. But thinking this way so misses a key consideration: we pay a significant price for context switching.

A familiar example is email. The total time we spend reading email is a productivity concern, but the larger concern for many of us is the frequency with which email causes us to interrupt our workflow. Knowing this, I made a brief attempt in 2008 to check email only once a day. Unfortunately, this approach would have violated too many of my peers’ expectations. I returned to status quo, reading my email (or at least scanning headers) as it arrives. Other messaging tools, such as instant messaging and Twitter, only add to the challenge of managing our personal communication flow.

Of course, what I really want is for my messaging tools to distinguish urgent messages from non-urgent ones, and to only interrupt my workflow for the former. I know that no system, whether based on manual filtering or algorithmic analysis, can make this subjective classification with 100% accuracy, but I’d certainly accept a handful of false positives in exchange for far fewer interruptions. I suspect I’m not alone.

Moreover, this approach extends beyond personal communications to more public ones, such as social media platforms and even web search. On one hand, the passing of time offers an opportunity to accumulate reliable content analysis; on the other hand, we don’t want to miss time-sensitive content just because the system waited too long to determine the content’s relevance to our information needs. Still, the low signal-to-noise ratio on social media platforms suggests to me that many information consumers would be amenable to a different tradeoff than the one we experience today.

What I’d really like to see is systems take advantage of the differences in users’ personal senses of urgency. Some examples:

• A widely broadcast email isn’t delivered all at once, but first goes to users with higher urgency settings. Because those users mark it as spam, the email is already marked as spam for users with lower urgency settings. Conversely, if enough high-urgency users mark it as important, then it may be sent to lower-urgency users sooner.
• High-urgency users frequently check news sites and blogs. If an article attract a threshold level of engagement from high-urgency users, then low-urgency users are notified. This approach could apply to general news or to news in a specific topic that the user follows.
• Same as above, but applied to activity feeds and based on engagement within your social network. But again, high-urgency users lead the way, seeing updates sooner but at the price of experiencing a noisier stream.

To some extent, our existing systems already approximate this approach. Mechanisms like favoriting and re-tweeting propagate signal from information scouts to their followers, as do algorithms that rank real-time information based on engagement. Still, as an information consumer, I’d appreciate an interface that explicitly and transparently adapts to my priorities, and that manages interruption of my workflow accordingly.

What do folks here think? Is information delayed tantamount to information denied? Or is time on our side, potentially offering us a better tradeoff than the one we experience today?