At the first HCIR workshop in 2007, Michael Stonebraker stood up in the middle of an open discussion session and told all assembled that we needed to be thinking about the deep web.
I don’t know how much the audience took heed of his call, but he certainly followed his own advice. He and Endeca alum Mark Watkins just launched Goby, a vertical search engine that exhorts you to “create your own adventure”. It’s fun–a sort of exploratory search for explorers. And it uses a deep web crawl to populate its index with semi-structured data.
Anyway, try it out! I’ve been in the private beta, but haven’t had the chance to see what they’ve been up to in the final stretch leading to the launch. You can also read more on Search Engine Land or CNET.
The Noisy Channel 
5 replies on “Goby Goes Deep”
Thanks Daniel – looking forward to hearing what The Noisy Community thinks!
LikeLike
Hi Mark,
Congrats. It really is nice and UI is very intuitive. I liked it. I have two suggestions: you can fill where field from visitor’s IP address and ability to close/hide floating map.
I am just a novice IR reader but can you give a little bit more information about the process? You have a field performers for musical events or duration and diffuculty for hiking trails and so on. Did you setup all these fields for every “What” manually or automatically? If answer is latter, did you get any help from other databases like Freebase? How did you extract structured data from pages? Custom formula for every site or a general method?
Thanks
LikeLike
I’ve only done one query so far, so I haven’t done a deep investigation. But I was very impressed. Usually services like these are great for urban metros, but break down very quickly when going out into rural areas. My query was for a rural area that I know well. It found lots of stuff to do in the next seven days. Impressive!
LikeLike
Hi Lee,
Thanks for the suggestions. We like the IP suggestion, although that’s not always as reliable as you might think, sometimes your ISP doesn’t actually know where you are. Another idea would be to just drop a cookie to preserve the location you searched for first, until you change it.
If you click on the drop-down to open up the map, if you then click on it again, it will close the map.
Much of the IP of the company is around the way that we find and categorize the content. The essence of our insight was that you can learn a lot about a page by how you get to it, either through a search form, or a browsing hierarchy, or through other means. The process is partly automated and partly editorial – a mix of software and people. But it’s cost-effective and scalable for the problem we are solving.
If you are in the Boston area, there is a potential that we’ll be giving a talk in the future about what we’re doing; I’ll certainly mention that if/when it gets scheduled.
LikeLike
Jeremy – thanks – we are big believers that there is an unmet need for helping find things to do in off-the-beaten-path places, rather than just “the top 25 cities” or something like that. Glad your first experience was good.
LikeLike