The Noisy Channel

 

Don’t believe everything you read in the New York Post

June 14th, 2009 · 36 Comments · General

Now this is the sort of publicity that even $100M can’t buy: the New York Post is reporting that, in response to Microsoft’s recent Bing launch, “FEAR GRIPS GOOGLE” (all caps in the original):

Sergey Brin is so rattled by the launch of Microsoft’s rival search engine that he has assembled a team of top engineers to work on urgent upgrades to his Web service.

I never imagined that anyone would get their technology news from the New York Post, but evidently it’s well read in the blogosphere. Techmeme reports the following articles as citing the New York Post article:

I know that the press loves a good fight, and in technology it’s hard to ask for a better pairing than Google and Microsoft. Moreover, I do think that Google should be paying attention to Microsoft’s positioning of Bing, regardless of how well Microsoft has delivered on that positioning. In any case, it makes sense for Google to keep close tabs on its competitors. After all, even a fraction of a percent of web search market share translates into millions–more than enough revenue to justify a few full-time employees.

Still, to assert that Google is gripped with fear stretches credibility, even for a tabloid. I don’t mean to suggest that Google is so self-confident as to be fearless. Google may well have reacted with fear when it looked like Microsoft would acquire Yahoo–in fact, some have suggested that Google’s proposed (but ultimately abandoned) advertising deal with Yahoo was a Machiavellian maneuver to scuttle the acquisition.

But, unless I’m missing something, Bing simply isn’t a threat to Google’s market dominance. If anyone should be concerned, it’s folks like Kayak who might lose some market share to Bing’s travel search–which seems to be generally acknowledged as Bing’s strongest vertical.

Personally, after being underwhelmed by Bing, I decided to try it for 2 weeks. I made it for about a week and a half, and you can see some of my commentary on Twitter. I stand by initial impression: it’s not bad, but it’s noticeably inferior to Google, and even parity is not enough to reverse the tide. Perhaps the tiny gain–or the slowdown in loss–that they will make in market share will justify their investment. But this is no revolution, and the Gevil Empire is not running scared.

36 responses so far ↓

  • 1 Christopher Rines // Jun 15, 2009 at 12:11 am

    “Bing simply isn’t a threat to Google’s market dominance. If anyone should be concerned, it’s folks like Kayak who might lose some market share to Bing’s travel search–which seems to be generally acknowledged as Bing’s strongest vertical”

    This is another example of why you’re blog is a must read in our space!

    Bing is not going to unseat Google’s core search offering but if I read the tea leaves correctly they’ve done what I’ve been saying they should do (not that they took my advice) and are attacking search from a different angle.

    Vertical Search is not new but a comprehensive engine supporting the top x verticals and working as a buying “Decision” support system has the potential of not only whacking other vertical plays, if it works Microsoft can gain a “large” market share in one of the highest revenue generating areas of the web.

    As well there’s a public mindset change that must take place no matter how good Bing is… That is the public needs to be convinced that even if they are using Google for 80% of their search tasks the other 20% of searching specific to vertical e-commerce type “answers” should be done at Bing.

  • 2 Daniel Tunkelang // Jun 15, 2009 at 9:02 am

    Indeed, I can see merit to chasing the top search tasks, unifying them through task-centric applications in a single site, and offering vanilla search as a fall-back for the remaining search needs. I’ll even break from my usual NLP skepticism and admit that NLP might even be helpful to route free-text queries to the right application, thus simplifying the interface so the user starts with a single, familiar search box.

    But I expect more from a task-centric application than what Microsoft has done with Bing. In general, I think search has focused so much on the “long tail” that is has neglected the short head.

  • 3 Roy Scribner // Jun 15, 2009 at 12:06 pm

    I started receiving traffic from Bing on June 1st. Of course, it’s minuscule compared to Google or even Yahoo, but it is on par with AOL. Two weeks is not enough data to judge traffic quality (or search engine effectiveness, depending on your perspective), but the results look decent, so far.

  • 4 Daniel Tunkelang // Jun 15, 2009 at 12:21 pm

    I think we’ll need to wait a bit longer to get a sense of the real Bing traffic numbers.

    http://www.netimperative.com/netimperative/news/2009/may/bing-traffic-drops-after-initial-surge-research

    http://weblogs.hitwise.com/robin-goad/2009/06/initial_bing_stats.html

  • 5 jeremy // Jun 15, 2009 at 2:24 pm

    Personally, after being underwhelmed by Bing, I decided to try it for 2 weeks. I made it for about a week and a half, and you can see some of my commentary on Twitter. I stand by initial impression: it’s not bad, but it’s noticeably inferior to Google, and even parity is not enough to reverse the tide.

    Could you do a post on exactly where you find Bing falling short? From your tweets, I see that you’re dissatisfied with spelling correction and other (what I would call peripheral) features, such as public transit directions. But in terms of core relevance, core interaction, core exploratoryness, core diversity, what are your problems? Does Bing get the same results as Google, but with the more relevant information ranked 4th and 5th rather than1st and 2nd? Does Bing miss certain pieces of relevant information altogether?

    In one tweet, you said Bing was worse on recall — what sort of information need did you have in that case? E.g. were you looking for digital cameras? Civil war historic battlefields? Options for various forms of alternative energy in your area?

    Basically, what was it about Bing that was inferior?

  • 6 jeremy // Jun 15, 2009 at 2:42 pm

    And speaking of public transit, does the city of New York itself not have a transit planner website? I lived in London, and they had the most fantastic trip planner:

    http://journeyplanner.tfl.gov.uk/

    It allows faceted options, such as whether you want to restrict your travel to certain types, such as buses, trains, tube, etc. Whether you have a bike, whether you have certain mobility restrictions. Unlike “simple-and-limited-functionality-interfaces-are-better” Google, it lets you mix and match walking and riding, i.e. you can be so flexible as to put a 12-minute walk into the middle of your tube and bus travel. Sometimes, if you are willing to walk a brisk 15 minutes, better connections are available, and the London journey planner will find those.

    I was car-less in London, and used it at least 2-3 times a week. It was absolutely better than any other transit planner I’ve seen, Google’s included. And it has been around longer than anything from Google, etc. — 5-6 years.

    New York doesn’t have something similar?

    I smell a business opportunity…

  • 7 Christopher Rines // Jun 15, 2009 at 3:44 pm

    “But I expect more from a task-centric application than what Microsoft has done with Bing. In general, I think search has focused so much on the “long tail” that is has neglected the short head.”

    That I whole heartily agree with. It’d be nice if Microsoft (or someone) went the whole way with a purposed UI & vertical style search is certainly prime target for HCIR.

    I’m betting there’s an internal struggle between the “old” Microsoft search & the “new” MS search team. The old team still believes they can beat Google at it’s own game while the new gang knows they have to do something different and as a consequence we get 1/2 (or maybe just a 1/4) a good idea meant to “please” everyone internally but doesn’t do as much as it could externally.

    The most telling thing from the NY Post article is not the “fear grips Google” hyperbole but the fact that Google’s brain trust is actually looking at Bing (which I believe) & sees compeditive ideas for the first time in a long time and they now know they have to react in some fashion.

    This reactionary position is somewhat new for them so it should be interesting to see how it plays out. They’re in the same boat with Twitter, caught off guard and trying to figure out the Google angle.

  • 8 Daniel Tunkelang // Jun 15, 2009 at 5:32 pm

    Lots to respond to!

    I see that you’re dissatisfied with spelling correction and other (what I would call peripheral) features, such as public transit directions.

    Those aren’t peripheral to me! My main use of Google Maps is for those public transit directions. And in 2009 I’ve come to take good spelling correction for granted. Maybe I’m spoiled, but I feel like Microsoft can’t afford to be sub par there. I suppose they can ditch public transit if they’re not after the NYC market.

    Re core relevance: this is typical: a search for tunkelang returns my blog as 10th. Google gets it in 2nd. Of course, that’s just one data point. But I tried the Blind Search to compare and rarely found Bing winning. Perhaps it’s just a matter of taste, in which case all I can conclude is that Google’s results are more to my taste.

    As for recall, I was specifically looking at news, and the difference is dramatic. For example,

    Bing: 6 results for endeca
    Google: 22 results endeca –and that’s restricted to the past month!

    Even worse when I search for tunkelang! I haven’t done a systematic study, but you can see why I’m not inclined to switch to Bing for news search. And their lack of blog search is hardly peripheral.

    Christopher, I suspect you’re right. Microsoft needs to supply more than half a loaf here, but perhaps what they’ve delivered is an unfortunate compromise. I’d like to see them be the task-oriented, HCIR-embracing Ungoogle, but I imagine that’s a hard sell to a management that has been focused on parity for years.

  • 9 jeremy // Jun 15, 2009 at 6:46 pm

    Those aren’t peripheral to me! My main use of Google Maps is for those public transit directions.

    No, what I mean is, that’s a specific vertical, not the main “general” search. And I didn’t mean to imply that it wasn’t important to you. I only meant to say that it wasn’t a generalist search, and so there is nothing stopping you from mixing and matching.. using Bing for your general web searches, and Google Maps for your NYC route planning (I still think London’s journey planner has everything else beat — there should be something similar for NYC).

    Same for news search. You’re comparing the news verticals. I’m curious about general search. Not all the “peripheral” offerings.

    And for general search, what I am trying to figure out is your intent (not just your taste) and why you are dissatisfied with the satisfaction of that intent. For example, do you mostly do navigational searches? Informational searches?

    What was the intent behind doing the [tunkelang] query? Were you specifically trying to find your blog? Were you trying to find your home page? Were you trying to find a method by which you could contact you (for example, through Twitter, Facebook, LinkedIn, Friendfeed, etc.)?

    Whatever happened to “don’t just guess the user’s intent — optimize communication” and “deemphasize the top ten documents — response is a set of documents”? Aren’t the sets of top 10 results in both Google and Bing equally relevant, from a set-based perspective?

    So I assume that you’re using both Google and Bing not to navigate. If you really were looking for your blog, typing [tunkelang blog] into both engines gets your answer. You wouldn’t have been trying to find your blog using the [tunkelang] query alone, would you? Because that would be more along the lines of the search engine trying to do intent guessing, rather than communication optimization.

    So which engine gives you better exploration and discovery possibilities?

    Remember your [cnn] query? Didn’t Bing seem much more exploratory? And I’ll be in Boston next month for a conference ;-) — try the more exploratory query [boston entertainment] on both engines. Google gives me a “maps” list of businesses, but with no explanation. Bing not only gives me a very similar maps-based list (see the “local” tab on the left), but also categorizes the results into theatre, comedy, nightclubs, restaurants, etc. That is much more interactive than Google. And it is conceptually much easier on my brain.. instead of having to look through an undifferentiated list of results (Google), Bing helps me formulate a mental model of the space.

    And I know I said above that we weren’t comparing verticals, but if you click the “news” tab on the left of the Bing [boston entertainment] results, you’ll get a a small set of entertainment-related, boston-related articles from newspapers. Not only is this same option not directly linked on Google, but even when I leave the main Google search engine and go over to the Google News vertical, and try both [boston entertainment] as well as ["boston entertainment] there, the results are orders of magnitude less relevant than the Bing ones.

    So doesn’t Bing do a much better job of thinking beyond single queries, and supporting refinement and exploration?

  • 10 Daniel Tunkelang // Jun 15, 2009 at 8:43 pm

    Point taken re: transit–though it is a real missed opportunity, considering that I’m probably not alone in using the same tool for directions and local search. Indeed, it’s a smooth transition for me to look up a business, find directions there, and then explore its surroundings (e.g., to find out where to grab a meal or drink nearby). Google gives me a better package here than Bing, and I’d say that this use of a search engine qualifies as “general web search”.

    I feel even more strongly that news sites and blogs are part of the “general” web. Much of the information I seek involves current developments (e.g., following the evolving Bing story), and it’s nice to restrict that search by source type.

    So lets go back to my examples, which I concede aren’t that meaningful without associated intent.

    When I look up a person, I hope to find the person’s home page (which might be a social network page these days), blog, resume, and publications (in a broad sense). Basically, I want a hyperlinked, current CV. Duck Duck Go and Kosmix probably comes closest to delivering this experience, because they actually try to deliver support for exploration. Neither Google nor Bing do, but Google gets slightly (but, in my experience, consistently) better coverage in its top ranked results.

    I grant that Bing tries a bit harder than Google to support refinement and exploration, though I think that a query like cnn is an unrepresentative softball. I think that a query like iran elections” is more typical, and Bing doesn’t offer much more than Google there, as far as refinement possibilities go. And I thought we already agreed that Google does better than Bing on news.

    I want the Bing that I saw in that “decision engine” video. I feel like Bing only delivers that kind of support for exploration on a handful of queries, and that for most queries it’s like Google, only worse. That’s why I stopped using it.

  • 11 jeremy // Jun 15, 2009 at 9:35 pm

    Point taken re: transit–though it is a real missed opportunity, considering that I’m probably not alone in using the same tool for directions and local search. Indeed, it’s a smooth transition for me to look up a business, find directions there, and then explore its surroundings (e.g., to find out where to grab a meal or drink nearby). Google gives me a better package here than Bing, and I’d say that this use of a search engine qualifies as “general web search”.

    (1) Go to Bing, try the query [boston entertainment]

    (2) Click the “Local” result (left hand side, in orange)

    (3) Refine/explore your results by rating, available payment methods (visa, diner’s club, etc.), and parking options (on site, valet, street, etc.)

    (4) Refine/explore your results by category. Select “music”.

    (5) Select the 2nd result, “Boston Center for the Arts”.

    See photos. Get reviews. Get driving directions. And as far as exploring the surroundings, not only do you have “Nearby Results” (see right hand side), but you also have one of the coolest things that Live Maps does, that no one else does: Bird’s Eye view. It’s up toward the top left. Click it. Get an overview sense of what the neighborhood looks like.. much better than Street View, and much better than satellite view. Use the “rotate” arrows to change your angle (N, S, E, W).

    Granted, it still doesn’t do transit planning for anything but the automobile. But it gosh darn does everything else, and with as much if not more faceted interaction as Google.

    When I look up a person, I hope to find the person’s home page (which might be a social network page these days), blog, resume, and publications (in a broad sense). Basically, I want a hyperlinked, current CV.

    Ok, so when I compare Bing’s [daniel tunkelang] with Google’s [daniel tunkelang], I get in order on Bing:

    (1) CMU page
    (2) LinkedIn
    (3) Blog
    (4) Crunchbase professional profile
    (5) Blog again
    (6) Backtype comments
    (7) Friendfeed

    On Google I get:
    (1) Blog
    (2) CMU
    (3) LinkedIn
    (4) Twitter
    (5) Crunchbase
    (6) Twitter again
    (7) WordPress comments

    Now, for evaluation, the satisfaction of your information need isn’t just any one piece of information. Your info need is satisfied only after you’ve accumulated *all* the information.

    So whether Google puts your blog first, rather than Bing’s third doesn’t matter, am I correct in saying that? Because you’re information need isn’t satisfied after only finding the blog. That’s a big part of what you mean when you talk about set retrieval, right? That the atomic unit of relevance isn’t one result.. it’s the whole set.

    So by the time we get to the third result on both engines, the set of results are exactly the same: we’ve found your blog, your home page (still at CMU? what gives? :-) and your LinkedIn “resume”.

    So the true measure of satisfaction, how many results you have to examine until every single piece of your information need is satisfied, is exactly the same in both cases: Three.

    Beyond that, both Bing and Google grab the Crunchbase listing. So the question really is: When searching for [daniel tunkelang], would you prefer seeing your tweets and your aggregated WordPress comments (Google), or your Friendfeed writings and your Backtype comments (Bing)?

    Now, maybe there, I would be wise to concede the point that its a matter of personal taste as to which of those two activities streams one wants to see. But they’re both activity streams, and by the time you get to the 7th result on both engines, there is as much comprehensiveness from both engines. Just a little twitter vs. friendfeed, backtype vs. wordpress differentiation.

    What I’m trying to do is not really disagree with you in any of this. I’m not trying to make any point that says either of us is right or wrong. I also want more decision engine than currently exists. What I’m after is trying to understand where is really breaks down. Because I personally have been pleasantly satisfied with Bing. If anything, I get more diversity by not using only Bing or only Google. But by rotating through both of them. Google misses Friendfeed (bad blood?) and WordPress. Bing misses Twitter and Backtype. By using both engines, I get it all, and my “set” results are that much richer. I would never stop using Bing, just as I would never stop using Google.

  • 12 Daniel Tunkelang // Jun 15, 2009 at 10:19 pm

    I’ll grant for the sake of argument that the results are comparable–maybe it really is just taste as far as the results themselves. Though here are some informational queries on my favorite topic of behavioral economics that are off the beaten path of Wikipedia entries and show Google to advantage:

    – hedonic arbitrage
    – anticipatory regret
    – preference reversal
    – hot hand
    – remembered vs experienced utility

    To be fair, Bing is often as good as Google for informational queries (where my information need is typically to quickly obtain a definition and the seminal results on the topic), but I haven’t found examples of informational queries where Bing succeeds and Google fails.

    Engines like Duck Duck Go, Kosmix, and Cuil tempt me with their support for exploration. Bing doesn’t offer me enough to tempt me. So it mostly feels like a Google clone with quality that is at best comparable, and at worst inferior–and falling short some of Google’s nicer bells and whistles that I actually use with some regularity (spelling correction, transit for maps, news, blogs, scholarly articles, books). I don’t say Bing needs to achieve parity across the board. But they do need to offer substantial differentiation for me to overlook small differences.

  • 13 jeremy // Jun 16, 2009 at 5:22 am

    - hedonic arbitrage
    – anticipatory regret
    – preference reversal
    – hot hand
    – remembered vs experienced utility

    Ok, I tried the first two queries here as well, and the Bing results were absolutely terrible. I knew there was no way they could purposely be that bad. It had to be a simple mistake. I thought a minute, and sure enough: Try the query again, with quotation marks. Look at the difference in Bing results for this:

    [anticipatory regret]

    ..versus..

    ["anticipatory regret"]

    So it’s just a little parsing error, which I’m sure Bing will fix. It’s only been around a week, after all. :-)

    Maybe I don’t know enough about the subject, but the ‘corrected’ Bing results seem just as relevant, if not just as important, because of their diversity from the Google biases.

    Both Google and Bing find the “changingminds.org” and “marketingcomet.typepad.com” sites. And Google is good.. it gets a Springer Verlag journal article and pulls back a book from Google Books. But Bing is also good: You said that these information needs were off the beaten path of Wikipedia entries, but Bing finds a Wikipedia entry on simulation heuristics in which the concept of anticipatory regret plays an important role. It’s almost a reverse of your typical search, where you’re more often looking for pages “about” anticipatory regret. Bing helps you find a page that is “about” something else, but in which anticipatory regret plays an interesting part. Surely this is useful to your study of the topic? And I don’t find that same result until rank 19 on Google. Same with an article from the Harvard Business School.. Bing finds it at rank 5. Google finds it at rank 37. (Yes, I looked.)

    I don’t know enough about the topic to know how truly useful each set of results is or isn’t. But they seem awfully similar to me in some respects (30% overlap in the top 10) and awfully different, but equally relevant, in other respects. Google’s Springer Verlag result is certainly good, but so is Bing’s Harvard Business School result.

    You really see an advantage to Google on this, though? Scanning the top 10 results of each, my holistic impression is that Google is tending to find more links from the academic and clinical psychology domain, whereas Bing is finding more links from the academic and practical business domain. Are the psychological results really the better ones to read?

  • 14 Daniel Tunkelang // Jun 16, 2009 at 8:09 am

    Quoting the phrase certainly helps Bing on the anticipatory regret query (let’s call it parity, though I personally like Google’s academic slant better), but shouldn’t a modern search engine be smart enough to figure out that a multi-word query might be worth treating as a phrase? And, consider this comparison:

    http://www.google.com/search?q=anticipatory+regret+-%22anticipatory+regret%22+-%22anticipatory+grief%22

    http://www.bing.com/search?q=anticipatory+regret+-%22anticipatory+regret%22+-%22anticipatory+grief%22

    Just goes to show that quoting phrases is a dangerous game. A more severe example is my last one: remembered vs experienced utility.

    Perhaps Bing, in the hands of a user who uses trial and error to create multiple variations of a query to see which one works, is comparable to Google. But failure to intelligently handle multi-word phrases is a more serious infraction than sub-par spelling correction.

  • 15 jeremy // Jun 16, 2009 at 1:54 pm

    but shouldn’t a modern search engine be smart enough to figure out that a multi-word query might be worth treating as a phrase?

    Sure. But Bing is a week old. Give ‘em time to work out a few kinks. It was years before Google got my [the the] query right. Y’know, the band?

    If you want to talk dumb, non-modern search engines, try the Google query [laptop computers -dell].

    There is some risk with this example, as the advertising that they show rotates a bit. But I tried the aforementioned query three times, and every time got a supposedly “relevant” ad for “Low-Cost Dell Laptops” linked to the “dell.com” domain at the very top of the ranked list. Not to mention a scattering of ads containing “dell” on the right hand side.

    Shouldn’t a modern search engine also be smart enough to know when I’ve set a “-dell” filter?

    I’d say that’s even a more serious infraction than either missing quotes or sub-par spelling. :-)

    Because this is fun, try the query [endangered species -"endangered species"] on both search engines. In that particular case, I think Bing pulls ahead. Google mangles some of my word order. Bing does a better job in preserving it. (This is what we called ordered vs. unordered windows, back at UMass, 17 years ago..)

    And I agree with you on one thing: On that “anticipatory regret” query, it did seem that Google had a more academic results set than Bing — something I also tend to prefer / agree with. But not everyone. And so one of two things needs to happen: Either people need to keep rotating their search engine usage, to be able to understand when one of their own queries might work better on one engine vs. the other, or else all engines need to open up and become less transparent, and give the user a way of saying, “y’know, for this query, I want more academic results…but for this other query, I want more popsci results”. I personally have both types of needs, for different topics. And if one engine is going to consistently prefer one type of information over the other, then that is a real hindrance to my information seeking. Wouldn’t you say?

  • 16 Daniel Tunkelang // Jun 16, 2009 at 3:03 pm

    The Bing brand may be new, Microsoft has been working on web search for over a decade. So I don’t cut them slack for being rookies.

    As for Google not respecting negation for ads, I have the feeling that is an industry standard, e.g., http://www.bing.com/search?q=laptop+computers+-dell. Google’s claims of relevant ads notwithstanding, I don’t think most users expect to have control over coercive advertising. That’s what ad blockers are for!

    To your point about academic vs. non-academic results, it may well be that Google has a bias toward the former that I perceive as better quality simply because my information needs often have that same bias.

    But, even as I was writing this comment, I was listening to Pandora and a unfamiliar song came on that sounded like The Weepies. I looked to see that it was by Deb Talan, a name I didn’t recognize. So I looked her up on Google and Bing. Google quickly gave me what I wanted to know–namely, that she’s part of The Weepies and not someone just ripping off their style. Bing could have gotten me there too, but not as smoothly. Yes, just one (more) data point, but it’s representative of my experience. Perhaps I’m just not cool enough to be a Bing person. :-)

  • 17 jeremy // Jun 16, 2009 at 3:32 pm

    The Bing brand may be new, Microsoft has been working on web search for over a decade. So I don’t cut them slack for being rookies.

    No, of course they’re not rookies. But what I am saying is that Bing is a significant upgrade from anything that came before. And sometimes there are glitches in the transition from the old system to the new. By way of analogy, sometimes the device drivers that you had in the old OS stop working under the new OS. That doesn’t mean that they’re rookies when they release the new OS. That just means that there are a few kinks to work out, still.

    Will the improvement come as a monthly hotfix? Or as an annual service pack? I don’t know. But Google is improving and changing all the time.. why not give Bing the same benefit of the doubt, and not simply stop using them altogether after only 1.5 weeks?

    If I had stopped using Google, because they consistently failed on my [the the] query for years (not just for 1.5 weeks), I’d be a poorer information worker for it.

    BTW, I tried both the queries [laptop computers] and [laptop computers -dell] on Yahoo. The former returned Dell ads. The latter was overly cautious, and didn’t return any ads at all. I tried it three times, both with and without the “-dell”, just to make sure it wasn’t a fluke. It’s not. Adding the “-dell” filter to Yahoo removes the ads.

    Looks like it’s not an industry standard, and Yahoo is the only entity doing the right thing, here. Yahoo 1, Google 0, Bing 0.

  • 18 Daniel Tunkelang // Jun 16, 2009 at 3:38 pm

    Perhaps I’m being too harsh. I’m certainly planning to keep track of Bing, even if I only use it for evaluation purposes. If they can improve significantly in a few months, they may yet win my traffic.

  • 19 jeremy // Jun 16, 2009 at 3:43 pm

    To your point about academic vs. non-academic results, it may well be that Google has a bias toward the former that I perceive as better quality simply because my information needs often have that same bias.

    But is every single query that you do necessarily best satisfied by academic results? Suppose you just saw a Discover Channel program on super-volcanoes, such as the one underneath Yellowstone. You want to learn a little more about it — is the best resource for you the Journal of Geological Science? A recent PhD dissertation on the topic? Or is it maybe a popular science magazine article?

    Let me hazard a guess that it’s the latter.

    So the question behind my point remains: What do you (the generic you), the information seeker, do, when your search engine of choice only gives you the scientific, but not the popular, or vice versa?

    There is nothing you can do, other than rotate your search engine usage. Right?

  • 20 Daniel Tunkelang // Jun 16, 2009 at 3:51 pm

    Of course, that’s an exaggeration. I’d say that Bing, Google, and Yahoo return mixes of scientific and popular results, and it’s more a question how where they have their sweet spot. My point is simply that, for the cases where I notice a difference between the results from Google and Bing–such as cases where the former’s skew towards academic results kicks in–that difference tends to favor Google for me. That may well be because my needs are disproportionately academic.

    I do rotate my search engine usage as a matter of intellectual curiosity and professional interest. I suspect most others will satisfice on a default rather than maintain such a rotation. I’ll be curious to see how many of them prefer Bing to Google.

  • 21 jeremy // Jun 16, 2009 at 3:55 pm

    I was listening to Pandora and a unfamiliar song came on that sounded like The Weepies. I looked to see that it was by Deb Talan, a name I didn’t recognize. So I looked her up on Google and Bing. Google quickly gave me what I wanted to know–namely, that she’s part of The Weepies and not someone just ripping off their style. Bing could have gotten me there too, but not as smoothly.

    I guess I didn’t have the same problem on Bing. When I tried the query on Bing, the very first result that came up was her discography: http://www.debtalan.com/sound.html

    And there are the weepies, right at the top, as part of Deb’s discography (©2003 the weepies (Deb Talan and Steve Tannen). Couldn’t be more clear than that.

    With Google on the other hand, I do get the “http://www.theweepies.com/” result ranked first. But neither the Google-based snippet, or the full page once I visit it, has Talan’s last name on it. Anywhere. So it’s not clear to me that Deb Talan is a part of the weepies, or if Google just matched the “deb” and not the “talan” keyword. Google’s best result was less helpfully-transparent.

    I’m definitely not one of the cool people, but I did find Bing much more helpful in this case. Are you getting different results than me?

  • 22 Daniel Tunkelang // Jun 16, 2009 at 4:04 pm

    Same results. But I had my confirmation without even clicking through a link–that was clear to me from seeing the “http://www.theweepies.com/” result ranked first.

    Incidentally, Google includes “the weepies” in its related searches, while Bing does not include them in their refinement / related search options. My point is more that, on real searches that I try in the course of information, Google consistently serves me either as well or better than Bing. I concede that it may be personal. But my point is that it’s consistent.

  • 23 jeremy // Jun 16, 2009 at 4:06 pm

    Yes, apologies, I also exaggerate when I say that a particular search engine “only” gives you scientific, or “only” gives you popular, results. There is definitely a skew, though. We both reached that conclusion independently.

    You really do prefer that skew almost every time, though? Truly? But.. don’t you work in industry? Wouldn’t you sometimes rather see what industry says, instead of academia? That Harvard Business School article was ranked 5th in Bing, and 37th in Google. You don’t mind that you miss it?

  • 24 jeremy // Jun 16, 2009 at 4:10 pm

    But I had my confirmation without even clicking through a link–that was clear to me from seeing the “http://www.theweepies.com/” result ranked first.

    But how was just seeing “the weepies” enough?

    And if just seeing that name was enough, well.. above the first Bing web page result is actually a little box with video results.. and a video with the title “The Weepies”. So if that name alone was your confirmation, wasn’t Bing at least equal — maybe even better, because you could click and actually see Deb Talan on video and audio, just in case?

    I don’t want to bend too far in defending Bing here. That’s not my intent. I just don’t see how Bing was “not as smooth”.

  • 25 jeremy // Jun 16, 2009 at 4:13 pm

    I just tried the Deb Talan result on Bing again, and got image results at the top, rather than video results.

    Looks like Bing is in a much heavier experimental phase right now.

    But the first time I did the [deb talan] query on Bing, I got the “the weepies” video result at the top.

  • 26 Daniel Tunkelang // Jun 16, 2009 at 4:13 pm

    I wish I could use a single engine that organized results and supported exploration! Instead, I have two ranked retrieval engines whose black boxes offer slightly different biases. The incremental value of switching among them doesn’t justify the cognitive load, so I satisfice.

    BTW, one of the reasons I am so harsh on Bing is that I was expecting them to be substantially different from Google. It’s precisely their similarity to Google that leads me to not think it’s worth consistently using both engines. That’s why I’m still more interested in Duck Duck Go and Kosmix.

  • 27 jeremy // Jun 16, 2009 at 4:15 pm

    I wish I could use a single engine that organized results and supported exploration!

    Well, you gots no disagreement with me, there ;-)

  • 28 Daniel Tunkelang // Jun 17, 2009 at 4:11 pm

    Wow, maybe the Bing traffic is real:

    http://comscore.com/Press_Events/Press_Releases/2009/6/Bing_Continues_to_Show_Growth_in_Search_Activity_According_to_comScore

    Or not:

    http://searchengineland.com/bing-comscore-sees-gains-compete-sees-same-21158

    Gotta love the wacky world of web analytics!

  • 29 Google Markets Itself | The Noisy Channel // Jun 17, 2009 at 8:40 pm

    [...] still don’t buy that Google is “gripped with fear“, but I agree with Danny Sullivan’s analysis that Google’s new “Explore [...]

  • 30 Bob Carpenter // Jun 25, 2009 at 2:07 pm

    I was shocked to see my vanity query [bob carpenter] not even on the first page of hits from Bing. My home page is #1 on Google and one of my books is in the top 10 along with one of my blog posts.

    But hang on. Should I be that highly ranked on Google? Bing’s number one and two his are Wikipedia pages for the ESPN sportscaster Bob Carpenter and U Delaware’s Bob Carpenter Center. Then there’s the NHL hockey player and guitarist for the Nitty Gritty Dirt Band.

    Maybe Google knows and I’m really more popular, and once Bing gets more traffic, it’ll know that, too, and I’ll percolate to the top.

    What’s interesting is that if the communities bifurcate, I can remain highly ranked on Google and all the techies will use Google. I have found Google to be better at just about anything programming or computer science related.

  • 31 Daniel Tunkelang // Jun 25, 2009 at 2:30 pm

    Ideally I’d want something like what Duck Duck Go does:

    http://duckduckgo.com/?q=michael+jordan

    Unfortunately, you’re no Michael Jordan:

    http://duckduckgo.com/?q=bob+carpenter

    Based on personal experience, I suggest you change your last name to one that is unique.

  • 32 jeremy // Jun 25, 2009 at 4:58 pm

    Bob,

    I completely agree with you; I find Google to be the best for technical queries. But I also find yahoo to be better for entertainment queries.

    That’s one of the disadvantages to not rotating your search engine usage, not trying to form a mental model of what tool to use when, and why. If you only use a single engine, and don’t develop the skills of knowing when to switch, your overall results will be poorer.

  • 33 Daniel Tunkelang // Jun 25, 2009 at 5:15 pm

    But what kills me is that you can’t skew any of these engines towards broad subject areas, like sports or technology.

    For example, all of the related searches for nlp on Bing relate to neuro linguistic programming. Google does better, but what you really want here is something like Duck Duck Go.

    I understand that won’t work on all queries. Broad-based categorization might be nice. For Bob Carpenter, I tried Clusty and Hakia. Not thrilled with either. Does anyone get this right?

  • 34 jeremy // Jun 25, 2009 at 5:56 pm

    Oh, I agree, Daniel. Too much emphasis is placed on the search engine doing it all for you, rather than on giving you control to do it yourself. You know better than anyone else what it is you want in that moment.

    The vaunted “personalization” solution also doesn’t quite work, because in addition to not giving you control, it assumes that what you want right now is similar to what you wanted in the past. If I usually follow sports, but then do a query for “ipod heart rate”, I don’t want a sports story about someone listening to the ipod during their workout. I want a piece of technology, e.g. the Nike+ipod device.

    Give. Users. Control.

    But until the major search engines do a much better job of that, the best we as users can do is develop an awareness of the biases that each of the engines have built into themselves. It’s not ideal, but it’s better than only sticking to one engine’s bias, alone.

  • 35 Daniel Tunkelang // Jun 25, 2009 at 7:52 pm

    Fair enough. I wonder how hard it would be to build generic query refinement modules, even something as crude as ANDing a query with a long OR of broad terms from a category. Of course, what I really don’t know is how a set retrieval technique like that plays with web search engines’ relevance ranking algorithms. I couldn’t find anything useful in the documentation. :-)

  • 36 jeremy // Jun 26, 2009 at 12:58 am

    I couldn’t find anything useful in the documentation.

    Do a web search for the documentation. It’s 1001st result.

Clicky Web Analytics