Don’t believe everything you read in the New York Post

“Bing simply isn’t a threat to Google’s market dominance. If anyone should be concerned, it’s folks like Kayak who might lose some market share to Bing’s travel search–which seems to be generally acknowledged as Bing’s strongest vertical”

This is another example of why you’re blog is a must read in our space!

Bing is not going to unseat Google’s core search offering but if I read the tea leaves correctly they’ve done what I’ve been saying they should do (not that they took my advice) and are attacking search from a different angle.

Vertical Search is not new but a comprehensive engine supporting the top x verticals and working as a buying “Decision” support system has the potential of not only whacking other vertical plays, if it works Microsoft can gain a “large” market share in one of the highest revenue generating areas of the web.

As well there’s a public mindset change that must take place no matter how good Bing is… That is the public needs to be convinced that even if they are using Google for 80% of their search tasks the other 20% of searching specific to vertical e-commerce type “answers” should be done at Bing.

LikeLike

I started receiving traffic from Bing on June 1st. Of course, it’s minuscule compared to Google or even Yahoo, but it is on par with AOL. Two weeks is not enough data to judge traffic quality (or search engine effectiveness, depending on your perspective), but the results look decent, so far.

LikeLike

Personally, after being underwhelmed by Bing, I decided to try it for 2 weeks. I made it for about a week and a half, and you can see some of my commentary on Twitter. I stand by initial impression: it’s not bad, but it’s noticeably inferior to Google, and even parity is not enough to reverse the tide.

Could you do a post on exactly where you find Bing falling short? From your tweets, I see that you’re dissatisfied with spelling correction and other (what I would call peripheral) features, such as public transit directions. But in terms of core relevance, core interaction, core exploratoryness, core diversity, what are your problems? Does Bing get the same results as Google, but with the more relevant information ranked 4th and 5th rather than1st and 2nd? Does Bing miss certain pieces of relevant information altogether?

In one tweet, you said Bing was worse on recall — what sort of information need did you have in that case? E.g. were you looking for digital cameras? Civil war historic battlefields? Options for various forms of alternative energy in your area?

Basically, what was it about Bing that was inferior?

LikeLike

“But I expect more from a task-centric application than what Microsoft has done with Bing. In general, I think search has focused so much on the “long tail” that is has neglected the short head.”

That I whole heartily agree with. It’d be nice if Microsoft (or someone) went the whole way with a purposed UI & vertical style search is certainly prime target for HCIR.

I’m betting there’s an internal struggle between the “old” Microsoft search & the “new” MS search team. The old team still believes they can beat Google at it’s own game while the new gang knows they have to do something different and as a consequence we get 1/2 (or maybe just a 1/4) a good idea meant to “please” everyone internally but doesn’t do as much as it could externally.

The most telling thing from the NY Post article is not the “fear grips Google” hyperbole but the fact that Google’s brain trust is actually looking at Bing (which I believe) & sees compeditive ideas for the first time in a long time and they now know they have to react in some fashion.

This reactionary position is somewhat new for them so it should be interesting to see how it plays out. They’re in the same boat with Twitter, caught off guard and trying to figure out the Google angle.

LikeLike

Those aren’t peripheral to me! My main use of Google Maps is for those public transit directions.

No, what I mean is, that’s a specific vertical, not the main “general” search. And I didn’t mean to imply that it wasn’t important to you. I only meant to say that it wasn’t a generalist search, and so there is nothing stopping you from mixing and matching.. using Bing for your general web searches, and Google Maps for your NYC route planning (I still think London’s journey planner has everything else beat — there should be something similar for NYC).

Same for news search. You’re comparing the news verticals. I’m curious about general search. Not all the “peripheral” offerings.

And for general search, what I am trying to figure out is your intent (not just your taste) and why you are dissatisfied with the satisfaction of that intent. For example, do you mostly do navigational searches? Informational searches?

What was the intent behind doing the [tunkelang] query? Were you specifically trying to find your blog? Were you trying to find your home page? Were you trying to find a method by which you could contact you (for example, through Twitter, Facebook, LinkedIn, Friendfeed, etc.)?

Whatever happened to “don’t just guess the user’s intent — optimize communication” and “deemphasize the top ten documents — response is a set of documents”? Aren’t the sets of top 10 results in both Google and Bing equally relevant, from a set-based perspective?

So I assume that you’re using both Google and Bing not to navigate. If you really were looking for your blog, typing [tunkelang blog] into both engines gets your answer. You wouldn’t have been trying to find your blog using the [tunkelang] query alone, would you? Because that would be more along the lines of the search engine trying to do intent guessing, rather than communication optimization.

So which engine gives you better exploration and discovery possibilities?

Remember your [cnn] query? Didn’t Bing seem much more exploratory? And I’ll be in Boston next month for a conference 😉 — try the more exploratory query [boston entertainment] on both engines. Google gives me a “maps” list of businesses, but with no explanation. Bing not only gives me a very similar maps-based list (see the “local” tab on the left), but also categorizes the results into theatre, comedy, nightclubs, restaurants, etc. That is much more interactive than Google. And it is conceptually much easier on my brain.. instead of having to look through an undifferentiated list of results (Google), Bing helps me formulate a mental model of the space.

And I know I said above that we weren’t comparing verticals, but if you click the “news” tab on the left of the Bing [boston entertainment] results, you’ll get a a small set of entertainment-related, boston-related articles from newspapers. Not only is this same option not directly linked on Google, but even when I leave the main Google search engine and go over to the Google News vertical, and try both [boston entertainment] as well as [“boston entertainment] there, the results are orders of magnitude less relevant than the Bing ones.

So doesn’t Bing do a much better job of thinking beyond single queries, and supporting refinement and exploration?

LikeLike

Point taken re: transit–though it is a real missed opportunity, considering that I’m probably not alone in using the same tool for directions and local search. Indeed, it’s a smooth transition for me to look up a business, find directions there, and then explore its surroundings (e.g., to find out where to grab a meal or drink nearby). Google gives me a better package here than Bing, and I’d say that this use of a search engine qualifies as “general web search”.

(1) Go to Bing, try the query [boston entertainment]

(2) Click the “Local” result (left hand side, in orange)

(3) Refine/explore your results by rating, available payment methods (visa, diner’s club, etc.), and parking options (on site, valet, street, etc.)

(4) Refine/explore your results by category. Select “music”.

(5) Select the 2nd result, “Boston Center for the Arts”.

See photos. Get reviews. Get driving directions. And as far as exploring the surroundings, not only do you have “Nearby Results” (see right hand side), but you also have one of the coolest things that Live Maps does, that no one else does: Bird’s Eye view. It’s up toward the top left. Click it. Get an overview sense of what the neighborhood looks like.. much better than Street View, and much better than satellite view. Use the “rotate” arrows to change your angle (N, S, E, W).

Granted, it still doesn’t do transit planning for anything but the automobile. But it gosh darn does everything else, and with as much if not more faceted interaction as Google.

When I look up a person, I hope to find the person’s home page (which might be a social network page these days), blog, resume, and publications (in a broad sense). Basically, I want a hyperlinked, current CV.

Ok, so when I compare Bing’s [daniel tunkelang] with Google’s [daniel tunkelang], I get in order on Bing:

(1) CMU page
(2) LinkedIn
(3) Blog
(4) Crunchbase professional profile
(5) Blog again
(6) Backtype comments
(7) Friendfeed

On Google I get:
(1) Blog
(2) CMU
(3) LinkedIn
(4) Twitter
(5) Crunchbase
(6) Twitter again
(7) WordPress comments

Now, for evaluation, the satisfaction of your information need isn’t just any one piece of information. Your info need is satisfied only after you’ve accumulated *all* the information.

So whether Google puts your blog first, rather than Bing’s third doesn’t matter, am I correct in saying that? Because you’re information need isn’t satisfied after only finding the blog. That’s a big part of what you mean when you talk about set retrieval, right? That the atomic unit of relevance isn’t one result.. it’s the whole set.

So by the time we get to the third result on both engines, the set of results are exactly the same: we’ve found your blog, your home page (still at CMU? what gives? 🙂 and your LinkedIn “resume”.

So the true measure of satisfaction, how many results you have to examine until every single piece of your information need is satisfied, is exactly the same in both cases: Three.

Beyond that, both Bing and Google grab the Crunchbase listing. So the question really is: When searching for [daniel tunkelang], would you prefer seeing your tweets and your aggregated WordPress comments (Google), or your Friendfeed writings and your Backtype comments (Bing)?

Now, maybe there, I would be wise to concede the point that its a matter of personal taste as to which of those two activities streams one wants to see. But they’re both activity streams, and by the time you get to the 7th result on both engines, there is as much comprehensiveness from both engines. Just a little twitter vs. friendfeed, backtype vs. wordpress differentiation.

What I’m trying to do is not really disagree with you in any of this. I’m not trying to make any point that says either of us is right or wrong. I also want more decision engine than currently exists. What I’m after is trying to understand where is really breaks down. Because I personally have been pleasantly satisfied with Bing. If anything, I get more diversity by not using only Bing or only Google. But by rotating through both of them. Google misses Friendfeed (bad blood?) and WordPress. Bing misses Twitter and Backtype. By using both engines, I get it all, and my “set” results are that much richer. I would never stop using Bing, just as I would never stop using Google.

LikeLike

– hedonic arbitrage
– anticipatory regret
– preference reversal
– hot hand
– remembered vs experienced utility

Ok, I tried the first two queries here as well, and the Bing results were absolutely terrible. I knew there was no way they could purposely be that bad. It had to be a simple mistake. I thought a minute, and sure enough: Try the query again, with quotation marks. Look at the difference in Bing results for this:

[anticipatory regret]

..versus..

[“anticipatory regret”]

So it’s just a little parsing error, which I’m sure Bing will fix. It’s only been around a week, after all. 🙂

Maybe I don’t know enough about the subject, but the ‘corrected’ Bing results seem just as relevant, if not just as important, because of their diversity from the Google biases.

Both Google and Bing find the “changingminds.org” and “marketingcomet.typepad.com” sites. And Google is good.. it gets a Springer Verlag journal article and pulls back a book from Google Books. But Bing is also good: You said that these information needs were off the beaten path of Wikipedia entries, but Bing finds a Wikipedia entry on simulation heuristics in which the concept of anticipatory regret plays an important role. It’s almost a reverse of your typical search, where you’re more often looking for pages “about” anticipatory regret. Bing helps you find a page that is “about” something else, but in which anticipatory regret plays an interesting part. Surely this is useful to your study of the topic? And I don’t find that same result until rank 19 on Google. Same with an article from the Harvard Business School.. Bing finds it at rank 5. Google finds it at rank 37. (Yes, I looked.)

I don’t know enough about the topic to know how truly useful each set of results is or isn’t. But they seem awfully similar to me in some respects (30% overlap in the top 10) and awfully different, but equally relevant, in other respects. Google’s Springer Verlag result is certainly good, but so is Bing’s Harvard Business School result.

You really see an advantage to Google on this, though? Scanning the top 10 results of each, my holistic impression is that Google is tending to find more links from the academic and clinical psychology domain, whereas Bing is finding more links from the academic and practical business domain. Are the psychological results really the better ones to read?

LikeLike

but shouldn’t a modern search engine be smart enough to figure out that a multi-word query might be worth treating as a phrase?

Sure. But Bing is a week old. Give ’em time to work out a few kinks. It was years before Google got my [the the] query right. Y’know, the band?

If you want to talk dumb, non-modern search engines, try the Google query [laptop computers -dell].

There is some risk with this example, as the advertising that they show rotates a bit. But I tried the aforementioned query three times, and every time got a supposedly “relevant” ad for “Low-Cost Dell Laptops” linked to the “dell.com” domain at the very top of the ranked list. Not to mention a scattering of ads containing “dell” on the right hand side.

Shouldn’t a modern search engine also be smart enough to know when I’ve set a “-dell” filter?

I’d say that’s even a more serious infraction than either missing quotes or sub-par spelling. 🙂

Because this is fun, try the query [endangered species -“endangered species”] on both search engines. In that particular case, I think Bing pulls ahead. Google mangles some of my word order. Bing does a better job in preserving it. (This is what we called ordered vs. unordered windows, back at UMass, 17 years ago..)

And I agree with you on one thing: On that “anticipatory regret” query, it did seem that Google had a more academic results set than Bing — something I also tend to prefer / agree with. But not everyone. And so one of two things needs to happen: Either people need to keep rotating their search engine usage, to be able to understand when one of their own queries might work better on one engine vs. the other, or else all engines need to open up and become less transparent, and give the user a way of saying, “y’know, for this query, I want more academic results…but for this other query, I want more popsci results”. I personally have both types of needs, for different topics. And if one engine is going to consistently prefer one type of information over the other, then that is a real hindrance to my information seeking. Wouldn’t you say?

LikeLike

The Bing brand may be new, Microsoft has been working on web search for over a decade. So I don’t cut them slack for being rookies.

No, of course they’re not rookies. But what I am saying is that Bing is a significant upgrade from anything that came before. And sometimes there are glitches in the transition from the old system to the new. By way of analogy, sometimes the device drivers that you had in the old OS stop working under the new OS. That doesn’t mean that they’re rookies when they release the new OS. That just means that there are a few kinks to work out, still.

Will the improvement come as a monthly hotfix? Or as an annual service pack? I don’t know. But Google is improving and changing all the time.. why not give Bing the same benefit of the doubt, and not simply stop using them altogether after only 1.5 weeks?

If I had stopped using Google, because they consistently failed on my [the the] query for years (not just for 1.5 weeks), I’d be a poorer information worker for it.

BTW, I tried both the queries [laptop computers] and [laptop computers -dell] on Yahoo. The former returned Dell ads. The latter was overly cautious, and didn’t return any ads at all. I tried it three times, both with and without the “-dell”, just to make sure it wasn’t a fluke. It’s not. Adding the “-dell” filter to Yahoo removes the ads.

Looks like it’s not an industry standard, and Yahoo is the only entity doing the right thing, here. Yahoo 1, Google 0, Bing 0.

LikeLike

To your point about academic vs. non-academic results, it may well be that Google has a bias toward the former that I perceive as better quality simply because my information needs often have that same bias.

But is every single query that you do necessarily best satisfied by academic results? Suppose you just saw a Discover Channel program on super-volcanoes, such as the one underneath Yellowstone. You want to learn a little more about it — is the best resource for you the Journal of Geological Science? A recent PhD dissertation on the topic? Or is it maybe a popular science magazine article?

Let me hazard a guess that it’s the latter.

So the question behind my point remains: What do you (the generic you), the information seeker, do, when your search engine of choice only gives you the scientific, but not the popular, or vice versa?

There is nothing you can do, other than rotate your search engine usage. Right?

LikeLike

I was listening to Pandora and a unfamiliar song came on that sounded like The Weepies. I looked to see that it was by Deb Talan, a name I didn’t recognize. So I looked her up on Google and Bing. Google quickly gave me what I wanted to know–namely, that she’s part of The Weepies and not someone just ripping off their style. Bing could have gotten me there too, but not as smoothly.

I guess I didn’t have the same problem on Bing. When I tried the query on Bing, the very first result that came up was her discography: http://www.debtalan.com/sound.html

And there are the weepies, right at the top, as part of Deb’s discography (©2003 the weepies (Deb Talan and Steve Tannen). Couldn’t be more clear than that.

With Google on the other hand, I do get the “http://www.theweepies.com/” result ranked first. But neither the Google-based snippet, or the full page once I visit it, has Talan’s last name on it. Anywhere. So it’s not clear to me that Deb Talan is a part of the weepies, or if Google just matched the “deb” and not the “talan” keyword. Google’s best result was less helpfully-transparent.

I’m definitely not one of the cool people, but I did find Bing much more helpful in this case. Are you getting different results than me?

LikeLike

Yes, apologies, I also exaggerate when I say that a particular search engine “only” gives you scientific, or “only” gives you popular, results. There is definitely a skew, though. We both reached that conclusion independently.

You really do prefer that skew almost every time, though? Truly? But.. don’t you work in industry? Wouldn’t you sometimes rather see what industry says, instead of academia? That Harvard Business School article was ranked 5th in Bing, and 37th in Google. You don’t mind that you miss it?

LikeLike

I just tried the Deb Talan result on Bing again, and got image results at the top, rather than video results.

Looks like Bing is in a much heavier experimental phase right now.

But the first time I did the [deb talan] query on Bing, I got the “the weepies” video result at the top.

LikeLike

I wish I could use a single engine that organized results and supported exploration!

Well, you gots no disagreement with me, there 😉

LikeLike