Category: General

Thanks to Amir Michail for a tweet that alerted me to a technical report by Ingmar Weber, Stephen Robertson, and Milan Vojnovic on “Rethinking the ESP Game“.

The ESP Game is a human-based computation game designed by Luis Von Ahn to tag images. Part of the his motivation for developing the game was that image tagging was too hard for machines. Hence, he decided to make it fun for humans to volunteer their own labor to the cause.

But it turns out that, once humans have supplied some initial labels to the game, a machine can take over. Here is the abstract for the technical report:

The ESP Game was designed to harvest human intelligence to assign labels to images – a task which is still difficult for even the most advanced systems in image processing. However, the ESP Game as it is currently implemented encourages players to assign “obvious” labels, which are most likely to lead to an agreement with the partner. But these labels can often be deduced from the labels already present using an appropriate language model and such labels therefore add only little information to the system.

We present a language model which, given enough instances of labeled images as training data, can assign probabilities to the next label to be added. This model is then used in a program, which plays the ESP game without looking at the image. Even without any understanding of the actual image, the program manages to agree with the randomly assigned human partner on a label for 69% of all images, and for 81% of images which have at least one “off-limits” term assigned to them. We then show how, given any generative probabilistic model, the scoring system for the ESP game can be redesigned to encourage users to add less predictable labels, thereby leading to a collection of informative, high entropy tag sets. Finally, we discuss a number of other possible redesign options to improve the quality of the collected labels.

Daniel Lemire commented recently that spammers may help make AI a reality. It’s interesting to see how yesterday’s Turing Test has become today’s CAPTCHA, and the competition between humans and machines is making both smarter.

I’m an avid LinkedIn user and fancy myself an expert on search, so I was excited today to see that LinkedIn has officially launched its new search platform. I recommend you watch the four-minute video below to get an overview of the new features.

First, the good news. The interface is slick and streamlined as promised. Type-ahead works smoothly, though it is restricted to the names of your contacts. The in-page options for refining your query by changing the sort or adding parameters are a welcome improvement to their previous interface, which took you to another page to make query modifications. And the presentation of results is clean and effective. Finally, there is a “saved searches” feature that also acts as a running query to alert you to new results. This is a great feature for recruiters.

Now, the bad news. There is still no support for exploratory search. I get 1,198 results when I search for Endeca (your results will depend on your personal network). That’s far too many to look through if I’m trying to establish contacts at a company, and sorting by relevance or relationship strength has limited value. What I’d really like is an overview of those 1,198 people that I can explore–by location, job title, their present and past relationship to Endeca, etc. Faceted search would certainly be more helpful than the unguided parametric search they offer.

I don’t mean to damn LinkedIn with faint praise–this is a significant improvement from the experience they offered before. But I wish that they would recognize the importance of exploratory search in the context of a professional networking site. As it is, they are leaving so much on the table.

Today, I heard about two blog analysis tools. Ever the empiricist, I decided to try them out.

Let’s start with GenderAnalyzer, which claims it can determine the gender of a blog author.

Well, it thinks I’m probably male:

• We have strong indicators that https://thenoisychannel.com/ is written by a man (90%).

That’s close enough to spare me any major gender identity issues. Well, maybe. Let’s look at a few blogs that are written by women:

• Gwen Harris’s Taxonomy Watch:
  We have strong indicators that http://taxonomy2watch.blogspot.com/ is written by a man (95%).
• Essays by Danielle Fong:
  We think http://einfall.wordpress.com/ is written by a man (79%).
• Claudia Imhoff at BeyeNETWORK:
  We think http://www.b-eye-network.com/blogs/imhoff/ is written by a man (89%).

Perhaps GenderAnalyzer doesn’t appreciate women in science and technology. Or perhaps it isn’t doing much better than random.

On to Typealyzer, which claims to perform a Myers-Briggs Type Indicator (MBTI) personality test on a blog. Let’s set aside the unscientifc basis of the theory and try it out.

• The analysis indicates that the author of https://thenoisychannel.com/ is of the type: ISTJ.

ISTJ, huh. I realize that some of you have never met me, but I assure you than I am an off-the-chart extrovert. If these personality tests have any merit, I’m somewhere between an ENTJ and an ENFJ. And, while Typealizer has a disclaimer that “writing style on a blog may have little or nothing to do with a person´s self-percieved personality”, I assure you that the personality of my blog accurately reflects the personality of its author.

So I wouldn’t put too much stock in blog analysis tools. Perhaps these two aren’t the best examples of the genre. But for now I’d suggest they be used for entertainment purposes only.

In The Hitchhikers Guide to the Galaxy, Douglas Adams tells the story of how the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons protest the development of a computer called Deep Thought to provide the Answer to Life, the Universe, and Everything. Lunkwill and Fook, two of the philosophers, argue, “what’s the use of our sitting up half the night arguing that there may or may not be a God if this machine only goes and gives us his bleeding phone number the next morning?” 

Deep Thought addresses their concerns as follows:

“but the programme will take me a little while to run.”

Fook glanced impatiently at his watch.

“How long?” he said.

“Seven and a half million years,” said Deep Thought.

Lunkwill and Fook blinked at each other.

“Seven and a half million years …!” they cried in chorus.

“Yes,” declaimed Deep Thought, “I said I’d have to think about it, didn’t I? And it occurs to me that running a programme like this is bound to create an enormous amount of popular publicity for the whole area of philosophy in general. Everyone’s going to have their own theories about what answer I’m eventually to come up with, and who better to capitalize on that media market than you yourself? So long as you can keep disagreeing with each other violently enough and slagging each other off in the popular press, you can keep yourself on the gravy train for life. How does that sound?” 

Adams wrote these words in 1979, before there was an Internet, let alone a World Wide Web or an attention economy based on it. And yet Adams could have been writing a parable about Google and the search engine optimization (SEO) industry.

According to Wikipedia, “Search engine optimization (SEO) is the process of improving the volume and quality of traffic to a web site from search engines via “natural” (“organic” or “algorithmic”) search results.” More importantly, there is a cottage industry of SEO software tools and consultants, all ready to help you optimize your web site for a fee.

You might think that Google, the search engine whose traffic most SEO efforts are optimizing, might frown on the use of SEO. After all, Google’s number one tenet is “Focus on the user and all else will follow.” Nowhere in their list of “ten things” do I see mention of supporting a multi-billion dollar industry of companies and consultants trying to manipulate result ranking.

And yet Google’s official position on SEO is hardly one of censure. Here is an excerpt:

If you’re thinking about hiring an SEO, the earlier the better. A great time to hire is when you’re considering a site redesign, or planning to launch a new site. That way, you and your SEO can ensure that your site is designed to be search engine-friendly from the bottom up. However, a good SEO can also help improve an existing site.

Full disclosure: Endeca sells SEO / SEM tools, and our customers see great results from them. We pursue a white-hat goal of “ensuring that all your content is exposed to Web search engines the right way”.

So, is SEO good or bad? Today, it’s actually necessary. Google and other web search engines rely on SEO efforts to compensate for the limitations of their indexing. This is an example of where sites share in the responsibilitiy for contextualizing a user’s experience.

But the adversarial nature of SEO is surely suboptimal for all parties. Well, for all parties other than SEO consultants and vendors. As Deep Thought said, “So long as you can keep disagreeing with each other violently enough and slagging each other off in the popular press, you can keep yourself on the gravy train for life. How does that sound?”

It sounds pretty lame to me. But it’s what we get when the attention economy of the web centers around a black box approach to relevance.

I just found this page explaining how to opt out of the “online communication experiment” called TheGreatHatsby which randomly attempts to connect two strangers through AOL instant messenger (AIM) bots.

Here are the instructions on how to opt out of this social experiment:

You can recognize the Bot by having names that are <adjective>Trout, <adjective>Salmon or <adjective>Coho.

You can stop the messages by typing:

$optout

Then it will respond with:

OPERATOR: Are you sure you want to opt-out? If you do, you will *never* be contacted again on the account “<screenname>”. There is *no way* to opt back in and undo this.
If you are sure, type “$optout DADD”. Remember, this is permanent and irreversible!

Type what it asks:

$optout DADD

And you will recieve one final (hopefully) message:

OPERATOR: You have opted out. The account “<screnname>” will *never* be contacted again. Good bye!

I just went through this process myself. I will let readers know if any more Coho bots show up.

Some readers may be familiar with The Pirate Bay, a Swedish website that indexes and tracks BitTorrent files. Those readers may also be aware that the site, and the pro-piracy Piratbyrån organization that runs it, have had various incidents with law enforcement. Perhaps some folks are looking for a contingency plan in case The Pirate Bay is shut down.

Have no worries, Google is there for you. As Ernesto at TorrentFreak writes, you can now build your own BitTorrent search engine for free using Google’s App Engine. In fact, the V0rtex and TorrentTab sites have already done so.

I suspect this is an unintended consquence of releasing the App Engine, much as Google Images was never intended to be the world’s largest porn site. But I’m curious how anti-piracy organizations like the RIAA and MPAA will react.

In recent weeks, I’ve been hit several times by instant messenger bots. The bots are rather clever, and evidently started in 2006 as TheGreatHatsby:

TheGreatHatsby^[1][2] is an AIM bot which instigates conversations between pairs of AIM accounts. Its name is a play on words from the book The Great Gatsby. It is a relay bot that retrieves the list of most recent LiveJournal posts and obtains the AIM screen names of their authors. It then sends users the message “i say, old bean, have you seen my hat?” from one of its screen names.

Responses from users are forwarded by the bot to another one of the users similarly contacted. These two users are paired up, and any message which one of them sends to the bot will be forwarded to the other. Thus, if neither of the users is aware of TheGreatHatsby, they will each think that the other user contacted them, and that the other user’s screen name is the bot’s screen name.

Messages containing a user’s true screen name will have that screen name replaced with the bot’s screen name; similarly, if a message contains the bot’s screen name, it will be replaced by the screen name of the user receiving the message. This adds to the confusion, since copy-and-pasted chat logs will appear as though the users really were messaging each other directly.

Specifically, I’ve been hit by “Coho bots“, which are bots as described above with names of the form “<adjective>Coho”.

I had ignored the messages in the past, but I finally decided to respond to one. Even though I was unfamiliar with IM bots, I naturally assumed that an IM user named “FidgetyCoho” was a bot. So I was taken aback by the very human responses I received in response to my response. After my unwitting (and unamused) conversational partner and I each established that the other was human, I did my research and learned the above about Coho bots.

Between this experience and my recent bout with a focused comment spammer, I must say I’m impressed and disturbed by the increasing level of sophistication of social media spamming. Absent a robust means of establishing identify and a cultural will to use it, I only see things getting worse.

Normally I take down spam commenters. But a recent comment spammer posting with the name “lively” (claiming the unlilkely email address of lively@gmail.com) struck me as interesting enough to to leave as an example to readers. You can see the comment here.

I searched my logs and found that this spammers discovered my blog through a highly targeted search query:

http://www.google.com/search?q=inurl%3Asearchwiki+leave+a+comment

I don’t know if the spammer manually posted comments to the pages in the results or used a fully automated tool. I also wonder at the spammer’s breadth–I could imagine a fairly simple tool that takes a few parameters (name, email, the targeted search query, and the comment payload) and then crawls and spreads the spam.

This approach is technically trivial and potentially quite devastating–especially if spammers step it up a notch and start varying the comments a bit to avoid detection through duplication. I suppose we’ll see a lot more CAPTCHAs around if this approach catches on.

This week’s New York Times Magazine features an article by Clive Thompson about the Netflix Prize. The Netflix Prize, sponsored by the Netflix movie rental company, is perhaps the best marketing stunt I’ve seen in the history of machine learning:

The Netflix Prize seeks to substantially improve the accuracy of predictions about how much someone is going to love a movie based on their movie preferences. Improve it enough and you win one (or more) Prizes. Winning the Netflix Prize improves our ability to connect people to the movies they love.

The Netflix Prize has captured the imagination of many information retrieval and machine learning researchers, and I tip my hat to the folks at Netflix for inspiring researchers while pursuing their self-interest.

But, for all of the energy that the contest has generated, I have my doubts as to whether it focuses on the right problem. As I commented at hunch.net a few months ago:

what are the real problems we’re trying to help users address through recommendations. I know that, as a Netflix customer, I don’t get much out of the numerical ratings (or out of sorting by them). When a friend recommends a movie to me, he explains why he thinks I’ll like it. Not only is that far more informative, but it also allows me to interact with the process, since I can correct errors in his explanation (e.g., I like British comedy but not slapstick, and hence hate Monty Python) and thus arrive at different recommendations.

I understand that a simple model of assigning a number to each movie is very easy to measure and lends itself to a competition like the Netflix Prize. But, as an end-user, I question the practical utility of the approach, both in the specific context of movie recommendations and more generally to the problem of helping users find what they want / need.

I am gratified to see the New York Times article raises a similar concern:

As the teams have grown better at predicting human preferences, the more incomprehensible their computer programs have become, even to their creators. Each team has lined up a gantlet of scores of algorithms, each one analyzing a slightly different correlation between movies and users. The upshot is that while the teams are producing ever-more-accurate recommendations, they cannot precisely explain how they’re doing this. Chris Volinsky admits that his team’s program has become a black box, its internal logic unknowable.

Specifically, the off-beat film “Napoleon Dynamite” confounds many of the algorithms, because it is nearly impossible to predict whether and why someone will like it.

It’s a matter of debate whether black box recommendation engines are better than transparent ones. I’ve repeatedly made the case for transparency–whether for relevance or recommendations. But the machine learning community, much like the information retrieval community, generally prefers black box approaches, because the restriction of transparency adversely impact the accuracy of recommendations.

If the goal is to optimize one-shot recommendations, they are probably right. But I maintain that the process of picking a movie, like most information seeking tasks, is inherently interactive, and thus that transparency ultimately pays off. Then again, I have drunk the HCIR kool-aid.

Google launched SearchWiki today–I hadn’t realized it was pre-launch when I blogged about it a few weeks ago. Here is their official description:

Today we’re launching SearchWiki, a way for you to customize search by re-ranking, deleting, adding, and commenting on search results. With just a single click you can move the results you like to the top or add a new site. You can also write notes attached to a particular site and remove results that you don’t feel belong. These modifications will be shown to you every time you do the same search in the future. SearchWiki is available to signed-in Google users. We store your changes in your Google Account. If you are wondering if you are signed in, you can always check by noting if your username appears in the upper right-hand side of the page.

My first thought: who could possibly care about this feature? This feels more like bookmarking than personalizing search, and the feature raises all sorts of questions because of the dynamic nature of the search results.

But then it occurred to me that this feature is supposed to feel like bookmarking. At least if I understand the intent, the goal is not so much about a personalized search experience as it is personalized information management. Think “Stuff I’ve Seen” or “Information Scraps“. Google expects users to treat the results for particular search queries as static pages that can be manipulated and marked up.

I find this idea striking and a bit disconcerting, especially given that, at least in my experience, most web search results are anything but static. But I can see the potential appeal of wanting to stop the world so you can edit it. I’m not entirely sure what makes this approach a search “wiki”, though it is certainly reminiscent of Mahalo and Wikia Search. Incidentally, both of those seem to be struggling.

One incentive for Google to get users to adopt SearchWiki is that its users will increasingly invest effort in storing information on Google’s servers. And, because this information is tied to Google search results, there’s far more lock-in than from using Gmail and Google Apps, whose content can be ported to competing applications.

And, unlike Mahalo and Wikia Search, Google is starting with the lion’s share of the web search market. Even so, I’m skeptical that many users will see value in this feature. Lively lasted about six months. I’ll give SearchWiki a year.