All that said, I’d be in favor of open access. But SIGIR is hardly a profit-making enterprise. I assume that the money they’d lose from Digital Library subscriptions would have to come from somewhere else, like conference fees, which in large part means from authors. Well, from their employers and funding agencies.

We went around and around on the money-making aspect of journals with Computational Linguistics. In the end, the ACL is pitching in an additional $20K/year to make the journal free. That $20K/year pays for an admin assistant for the journal (1 day/week) and for copy-editing and typesetting/layout. For some reason journals will pay for assistants but not for editors or reviewers.

In the end, I think what swayed everyone was the experience of JMLR. It’s now the leading journal for machine learning, and part of that may be attributable to its open access policy.

Not to sound like a raving open-source lunatic, but the fact that an outfit like SIGIR’s making money off of the whole enterprise doesn’t make me want to review for them or submit papers.

And not to advocate this approach, but some open-access biology journals push the price off to the authors, such as Oxford’s Nucleic Acids Research.

I’m not surprised that the Digital Library is a money-maker–to me, it’s the main motivation for renewing my membership (not to mention paying extra for access to the library). I’m sure there would be broader readership if access to the Digital Library were free, but I’m not sure if authors (who would presumably be the folks pushing hardest for broader readership) would be willing to pay extra to make up for the lost revenue. This seems to be one of those rare cases where a lot of readers are willing to pay for content.

Incidentally, on the topic of the ACM Digital Library, I think they said during the SIGIR’08 Business Meeting that SIGIR actually makes more money off of the Digital Library than any other source (even compared to membership subscriptions and such). I would prefer that they do the same thing as the WWW Conference and post all papers online, however.

If you’re curious, I think I’m going to post a report on SIGIR’08 to the Stanford InfoBlog tomorrow or Wednesday, though I don’t think I’m going to talk about the business meeting. (It was pretty procedural—Should we have video? Where do we get money? How should we change reviewing given that we can’t have the Senior PC meet for longer than a day?).

The only paper I can’t find openly available online is “Real-time Automatic Tag Recommendation”. Most of the time, authors post papers on their home pages (or at worst will send copies of their papers on request), and the ACM doesn’t seem to intervene. I’m not sure how much the ACM counts on revenue from Digital Library subscriptions, but I imagine that making the proceedings open-access would put a dent in that revenue.

One way to make this happen is through reviewing. I only agree to review for venues that publish open access.

The Journal of Machine Learning Resarch and the United States National Institutes of Health are an inspiration. Here’s the ML vs. JMLR story and here’s the NIH open access policy.

The Computational Linguistics journal will be open access as of May 2009; the ACL conference proceedings have been open access. The ACL executive board and journal editor (Robert Dale) worked hard to make CL open access.

Fair criticism, and perhaps I’m betraying my enthusiasm at any work that pursues the problem of correlating system performance measures with user effectiveness. For example, I was psyched to see Stephen Robertson derive average precision from a user model, supporting the case many of us have been making that system performance measures need to be ultimately motivated by users.

In any case, while I’m in favor of considering a broader set of measures, I’m not sure that satisfaction is the most important one to optimize. I think user effectiveness at a task level is at least as important, and that recall-oriented tasks have been given short shrift in a community that over-emphasizes precision in the top-ranked results.

- Their user task is purely recall based and they prove that p@200 correlates better than MAP with user satisfaction. Of course, since the task they’ve tested correlates a lot with recall.

- They try to look at the correlation of various P@ – P@10, P@20 etc. with the time needed to save the first relevant documents and discover that P@10 correlates most of all(!). It was not hard to guess… And not to say, that there are specific measures for that, like Mean Reciprocal Rank.

I would be much more in favour of a paper that measures correlation of all possible measures with satisfaction of doing all possible tasks with a system.