Comments on: Free as in Freebase

By: Relational databases: are they obsolete?

Relational databases: are they obsolete? — Wed, 16 Sep 2009 20:25:58 +0000

[...] as performance is concerned, Stonebraker is obviously right: we are undergoing major changes. As pointed out by Daniel Tunkelang, you can store a lot of data in 32GB of RAM. Solid-state drives can be used to wipe out some IO [...]

By: Daniel Tunkelang

Daniel Tunkelang — Mon, 31 Aug 2009 16:04:38 +0000

In any case, comparison is tricky, since Freebase is a structured repository that isn’t storing large blocks of text–let alone images and video.

By: Daniel Lemire

Daniel Lemire — Mon, 31 Aug 2009 16:00:31 +0000

The uncompressed size of a wikipedia dump is over one terabyte. There are solid state drives with that capacity—though they are expensive. However, in *compressed* form, wikipedia can fit within 4GB and thus, could fit in my one-year-old laptop’s RAM.

Now that wikipedia has decided to accept video clips, however, expect the size of the wikipedia dumps to go up significantly in the next few years.

Reference:
http://download.wikipedia.org/backup-index.html

By: Daniel Tunkelang

Daniel Tunkelang — Mon, 31 Aug 2009 15:02:43 +0000

I suspect the best comparable is the very modest size of Wikipedia–since Freebase, at least from my perspective, is trying to be a Wikipedia of structured data.

http://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia

By: Daniel Lemire

Daniel Lemire — Mon, 31 Aug 2009 14:56:04 +0000

As a basis of comparison for the 32GB figure… According to Jim Gray, you can store everything you read in a year in 25 MB.

Reference:
http://www.daniel-lemire.com/blog/archives/2006/10/26/what-is-infinite-storage/

By: Daniel Tunkelang

Daniel Tunkelang — Mon, 31 Aug 2009 14:30:02 +0000

I didn’t mean to imply otherwise. But I do think it’s an eye-opener for folks who are used to measuring open web content in petabytes. Trust me, I know from experience that putting data in memory isn’t a silver bullet when you need to do anything interesting with it.

By: Daniel Lemire

Daniel Lemire — Mon, 31 Aug 2009 13:52:50 +0000

Perhaps the most surprising revelation about Freebase was that all of their data fits in RAM on a 32G box (…). Their biggest challenge is collecting good data

Just in case one of your reader decides that we no longer need fancy database indexes or good engineers because RAM is so cheap… Know that it will take over 5 seconds to read the content of the memory—sequentially.

People have been designing RAM-based databases for indexing data such as XML for a while. The performance is not automagically acceptable.

By: Daniel Tunkelang

Daniel Tunkelang — Sat, 29 Aug 2009 23:21:29 +0000

Bob, I think it's worth noting that, had Google not been, um, inspired by Goto.com's revenue model of auctioning sponsored listings, it's not at all clear that Google would have succeeded. Remember: Google didn't introduce AdWords until 2000. It's not at all clear, that predicting Google's success back in the late 1990s would have been rational. Hindsight is 20/20.

By: ram

ram — Sat, 29 Aug 2009 23:15:09 +0000

Thanks Daniel for posting blog on Freebase. Hope it gains momentum like Wikipedia and manages to overcome the challenge of collecting the semantically enhanced data sets.

By: Bob Carpenter

Bob Carpenter — Sat, 29 Aug 2009 23:14:11 +0000

I just talked to Jamie and Robert recently, and was very impressed that Freebase was addressing the shared semantic problem through integration. Clearly they’ve already built something useful.

As to profitability or even revenue, I was wondering the same thing about Google during a visit back in the late 1990s — awesome search, crazy company (funhouse office, food better than most restaurants, programmers complaining about not getting massages quick enough), and no revenue stream in sight (at least to me).