Author: Daniel Tunkelang

Several people have asked me recently for advice on how to recruit for their tech startups. I’ve responded by digging out the following email that someone emailed me last year. I reproduce it in full here, minus the company name:

Subject: we just got Beatles Rock Band for the office and are looking for a vocalist !!

Good Afternoon,

I hope you don’t mind me reaching out to you, but came across your LinkedIn page and my interest is peaked, to say the least. I hope after reading this you feel the same.

If you’re unfamiliar with XXXXXX, we are a distinct small and agile team that functions as an incubated start-up funded by a larger organization. What we are working on is still kind of a secret but I can tell you that it’s focused on completely changing the way we find, consume, share, and manage content on the web today. We are focused on the growing importance of the real-time web and the concurrent need to reduce the noise. We are driven by a strong desire to deliver a better overall experience with a lot less effort required from our users.

Our office is extremely open and collegiate, and we are committed to letting ideas thrive above all else. We’re a very eclectic bunch of characters, but we all share a common commitment to taking whatever we do, fun or work, to the max. Some words that have been used to describe us are: passionate, fun, funny, innovative, contrarian, automagical, brilliant, academic, whimsical, and most importantly respectful. If you fit 3 or more of those descriptions, you might just have some of that magic we’re looking for.

If you’re interested in exploring this opportunity, please email me your resume and I’ll follow up with you ASAP, and have you come by meet the team some time soon.

Either way, I hope to hear from you!

Have a great weekend,

We too are BIG karaoke fans ( I read your website) , and as I said above we just got Beatles Rock Band for the office and are looking for a vocalist !!

Cheers,
XXXXX

I see this email is a poster child of how a startup should recruit. It’s well-written, funny, and shares enough about the opportunity to be an effective hook. Most importantly it’s *personal*. Starting from the subject line that made a great first impression, the email showed proof that the sender–a complete stranger–had taken time to get to know about me.

This is a strategy that does not scale arbitrarily–and that is the whole point. A startup that is building a small team needs to choose its prospective employees carefully and then go after those prospects with full force. If you really want to earn someone’s attention, you have to show that you’ve invested attention yourself. There’s no free lunch–if you want to send out a hundred emails like this one, you’ve got your work cut out for you! But no startup should be recruiting on such a massive scale, and the increase in yield justifies the additional per-candidate investment.

Of course, this principle applies beyond the narrow context of recruiting. Indeed, it is much like an attention bond mechanism: prove to me that you’ve invested in targeting me personally, and I’ll be more inclined to invest my attention in reading your message. Indeed, search advertising follows a similar principle. I still maintain that search is not advertising, but perhaps this aspect of negotiating a shared interest between messenger and messengee is a common thread.

A research study I like enough to have blogged about it a few times is Princeton sociologist Matt Salganik‘s dissertation work on music preferences and social contagion. For those unfamiliar with this work, here is the abstract of his Science article “Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market” (co-authored with Peter Dodds and Duncan Watts):

Hit songs, books, and movies are many times more successful than average, suggesting that “the best” alternatives are qualitatively different from “the rest”; yet experts routinely fail to predict which products will succeed. We investigated this paradox experimentally, by creating an artificial “music market” in which 14,341 participants downloaded previously unknown songs either with or without knowledge of previous participants’ choices. Increasing the strength of social influence increased both inequality and unpredictability of success. Success was also only partly determined by quality: The best songs rarely did poorly, and the worst rarely did well, but any other result was possible.

The result is hardly surprising to anyone familiar with the history of pop music. But I’m intrigued by the possibility that technology is simultaneously pulling music as a social phenomenon in two opposite directions.

On one hand, YouTube and social networks may actually be amplifying the positive feedback of music popularity. The recent story of YouTube sensation Greyson Chance (yes, a 13-year old with his own Wikipedia entry) becoming a national phenomenon in a couple of weeks attests to the power of social contagion. I don’t mean to take anything away from Chance’s talent, but I feel safe asserting that his talent was necessary but hardly sufficient to achieve his popular success.

On the other hand, Internet radio services like Pandora and Last.fm, despite their social features, offer the possibility of drastically reducing the effect of social influence. Both of these services require users to provide some representation of their musical tastes as initial inputs, whether by selecting preset stations or using particular artists or songs as seeds. Presumably those tastes are in large part the product of social influence. But the subsequent interaction between users and these services is relatively buffered from social influence. Users hear songs while listening privately through headphones–in many cases at work or while commuting. No one else is around when those users decide how to rate what they are listening to.

Granted, social context will always seep in–I don’t think I could give a thumbs-up to a Justin Bieber song even in the privacy of my own Pandora profile. But much of the music I discover is from artists I’ve never heard of–and thus evaluate without the explicit social influence of preconceptions about those artists.

As it turns out, I often discover after the fact that a number of the artists I like have achieved popular success. I can’t tell whether that reflects on their objective music quality, my own conformity of musical taste, or skew on the part of the recommendation system (cf. does everything sounds like Coldplay?). Still, I’m quite sure that I’m not favoring music based on prior knowledge of its popularity –for the most part, I don’t have that information at the time that I decide whether I like a song. Indeed, I hear new music almost exclusively through Pandora.

I don’t know how exceptional I am as a media consumer, but I suspect my case is increasingly common. Perhaps we are heading into a world where there will be a split between musical taste as social currency vs. musical taste as purely personal pleasure. It’s harder for me to imagine books or feature-length movies becoming so divorced from social context, if only because consuming them is a much larger and concentrated investment.

Still, I think it’s a big deal that this is happening in music. It’s a welcome counterpoint to the winner-take-all dynamic that has dominated the past decades of pop music. I can’t say that it will make the music industry more of a meritocracy–or that I even know what that would mean. But I think it’s a welcome step away from the caricature of conformity demonstrated by Salganik’s research.

I’m looking forward to attending SIGIR 2010 in a few weeks and particularly to the SimInt 2010 Workshop on the Automated Evaluation of Interactive Information Retrieval. I hope I get to see a little bit of the city of Geneva, but mostly I’m excited to spend the greater part of a week immersed in the global information retrieval community.

Of course I’ll blog about the conference, though I can’t promise it will be at quite the level of detail I managed last year. Also, I’m glad that SIGIR is continuing to have an industry track, and I am impressed with the program that David Harper and Peter Schäuble have put together. Needless to say, I’m glad to not have the stress of being an organizer this year! Though I’ll put in an early plug for CIKM 2011 in Glasgow, where I’ll be organizing the industry track with former co-worker Tony Russell-Rose.

Some SIGIR papers that caught my attention in the program:

• Predicting Search Frustration
  Henry Feild, James Allan (University of Massachusetts Amherst), Rosie Jones (Yahoo! Labs)
  (looks like a follow-up to the first two authors’ HCIR 2009 paper on Modeling Searcher Frustration)
• Relevance and Ranking in Online Dating Systems
  Fernando Diaz, Donald Metzler, Sihem Amer-Yahia (Yahoo! Labs)
• On Statistical Analysis and Optimization of Information Retrieval Effectiveness Metrics
  Jun Wang, Jianhan Zhu (University College London)
• Is the Cranfield Paradigm Outdated? (keynote)
  Donna Harman (NIST)
• Interactive Retrieval Based on Faceted Feedback
  Lanbo Zhang, Yi Zhang (University of California at Santa Cruz)
• Do User Preferences and Evaluation measures Line Up?
  Mark Sanderson, Monica Lestari Paramita, Paul Clough, Evangelos Kanoulas (University of Sheffield)
• Human Performance and Retrieval Precision Revisited
  Mark D. Smucker, Chandra Prakash Jethani (University of Waterloo)

As for the SimInt workshop, it aims “to explore the use of Simulation of Interactions to enable automated evaluation of Interactive Information Retrieval Systems and Applications.” I’m very excited about this attempt to bridge the gap between TREC/Cranfield and IIR/HCIR through simulation. Props to Leif Azzopardi, Kal Järvelin, Jaap Kamps, and Mark Smucker for organizing it!

If you’re planning to attend SIGIR, please give me a shout! I plan to be there for the entire conference, and you’ll probably find me at the Google booth during some of the coffee breaks.

Prediction markets were all the rage a few years ago, two of the most notable being the Iowa Electronic Market forecasting electoral results and the now defunct Tradesports offering a similar platform for betting on sports events. There was even a proposal to have the US government run a prediction market for terrorist attacks.

In a prediction market, any event with a quantifiable (e.g., binary) outcome can be converted into an asset. At any given time, the asset value corresponds to the market prediction of the probability of the outcome. Just as in any security market, participants determine the value through their buying and selling actions. In principle, this framework allows any event with a quantifiable outcome to be predicted by a marketplace.

But, at least from my vantage point, prediction markets have not had a broad impact on decision making, despite all of the “anys” in the previous paragraph. Outside of political forecasting and sports gambling (and of course finance itself), I’m not aware of any groups outside of academia that invest significantly in the use of  prediction markets. Sure, there’s the Hollywood Stock Exchange that applies the fantasy sports concept to the movie industry and even startup Empire Avenue that aspires to generalize this idea even further into an “online influence stock exchange”. Still, I think it’s safe to say that prediction markets have had limited traction to date.

Many people do, however, believe that we can harness the wisdom of crowds. In particular, we as consumers rely on reviews and recommendations to inform our decisions about what to buy, read, etc. Because those decisions have financial implications for sellers, the world of online reviews has an adversarial element, where review systems face manipulation by those who would shill their own products or services. As a result, it is never clear how much we as consumers should trust the reviews we read to be sincere, let alone useful.

Which brings me back to prediction markets. Unlike most venues for soliciting collective opinion, prediction markets offer a strong incentive for accuracy. Betting on whether readers will like a book is quite different than simply offering a review that asserts an opinion without any risk to the person making the assertion. It is possible to manipulate a prediction market (e.g., by flooding it with high bets), but research suggests that such manipulations are short-lived and in fact expose the manipulator to significant financial risk when the price re-stabilizes.

So why don’t we use prediction markets instead of relying on reviews and recommendations? Perhaps we should, and it’s just a matter of time until entrepreneurs build successful businesses around this idea. But I suspect that much of the value of user-generated content today comes from contributors not thinking in market terms. While using prediction markets could solve the problem of shill reviews, it might also scare off the altruists.

Still, it seems to me that we should look for more opportunities to incent accuracy. Even altruistic reviewers have an interest in establishing their credibility, at least if that credibility determines the propagation of they opinions they share (perhaps I’m conflating altruism with egotism). The challenge may be to implement a marketplace that deals in the social currency of reputation than the hard currency of cash–while avoiding the sort of virtual currency that many people see as meaningless.

Can we obtain the benefits of market dynamics and still take advantage of the less rational motivations that drive some of the best online reviews today? I hope there are people who feel incented to work on this problem!

Some previous posts for further reading:

• Thoughts About Online Reputation
• Payola? There’s An App For That!
• Are Links A Distraction? (please let me know if you like my attempt to keep links outside the main flow)

Eric Andersen called my attention to a post by Nick Carr entitled “Experiments in delinkification“, in which Carr argues that links embedded in text are distracting, and that we’re better off treating them like the footnotes they evolved from and putting them in a block at the end of the text. It’s an interesting piece, and I see the merits of his argument. Indeed, I remember trying to read a heavily annotated edition of Nabokov’s Lolita, and it was extremely hard to maintain the flow of reading the novel while turning every few seconds to read about every last entomology reference in the text.

Nonetheless, I feel that links supply context, and I’m a fan of keeping context nearby. Indeed, I find that clicking on a link incurs a much lower cognitive cost than flipping to the back of the book, searching for an endnote. I’ve had readers specifically thank me for including links to Wikipedia entries for technical terms. I assume those readers are fully capable of finding those Wikipedia entries themselves, but that they appreciate the convenience of the links.

Some of the commenters on Carr’s post suggest that we use technology to address this tension between preserving the reader’s focus and supplying nearby context. Specifically, we can use CSS and have a JavaScript button that toggles the link style between visible and invisible. I like the idea of handing readers control of the presentation style, though I still think it’s important to pick a sensible default. At the very least, a document should be self-contained so that a reader can choose if and when to look at the material it cites. The document should also give credit where it’s due, linking to the material it cites in a way that is visible to people and search engines. Beyond that, I think it’s really a matter of author style.

Still, I’m curious what folks here–especially long-time readers–think. Do I link so heavily that it’s distracting? Would it be easier to read my posts if the links were in a block at the end? I write for you, so please let me know how I can make this blog better. I don’t have the resources to conduct cognitive load experiments, but I’m very receptive to comments.

The other day, I received a surprise package in the mail: a copy of IBM researchers David Carmel and Elad Yom-Tov‘s newly published lecture on “Estimating the Query Difficulty for Information Retrieval“. I wasn’t even aware that this book was being written, so I’m especially appreciative of the publisher’s kindness to send me a copy.

If you liked Claudia Hauff‘s recent dissertation on “Predicting the Effectiveness of Queries and Retrieval Systems” (cf. my blog post on how “Not All Queries Are Created Equal“), then you’ll love this compact lecture that review the work on pre-retrieval and post-retrieval prediction of query performance. It covers query clarity, ranking robustness, query coherence, and much more.

I’m a big fan of the Morgan & Claypool series of Synthesis Lectures on Information Concepts, Retrieval, and Services, though I’m admittedly biased. Still, I think these books are an excellent way to get an overview of a subject, and Carmel and Yom-Tov’s book delivers wonderfully. For those not lucky enough to receive free copies in the mail, I recommend Amazon, which is selling it for less than $24.