I’ve been reading Einstein’s biography and he said something along the lines of it not being good for scientists to confine themselves to publishing papers because you ended up writing what you thought would be accepted rather than what is interesting to you. I find myself in agreement.

I submitted a paper about my phd thesis on conversational agents and it was rejected because the reviewers doubted very much that chatbots would be used in business. They were wrong, although I should have written a better and more convincing paper. I do wonder how subjective this reviewing process really is.

I agree with everything Daniel has said so far. I also think using video and the web would be better. I have had papers withdrawn because I couldn’t get to the conference for example. You need funds to travel half way across the world to present a paper. Shame to not publish just because of that, and so many conferences insist on you being there in person.

I feel that conferences need to be more inclusive as well. I know some very talented programmers with very solid and exciting ideas who wouldn’t know how to write a paper for submission. They are not academics and the papers must be of a high standard I agree, but can we make it more accessible? It appears to me that a video presentation, an online demo, an online discussion and these other mediums allow for these non-academics with plenty to offer the field to participate.

I think that peer review is of huge importance to us all, but at the same time rejecting 85% of papers is quite a lot when you think about it. I want to read all the rejections, and like Daniel I have no trouble dealing with the extra noise.

This is impractical but I’ll suggest it anyway:

Could we send through a proposal for a presentation, like an abstract for example and then as an organiser you have an idea of the types of projects and can form natural categories for this year. Duplicates can be coupled together an the authors notified and it can be suggested that they communicate together and maybe do a joint presentation of some kind (rather than reject 9 and accept 1 of them). Then once organised in groups or alone, people upload their presentations 2 weeks before the conference to give others time to view them.

The conference is all about discussion rather than presenting this way.

Wouldn’t you value a person who has answered an hour of pertinent questions from his/her peers rather than one that has talked at/to them for 30mins? More importantly, someone who during a discussion has been able to give more to the field?

Conferences become about everyone, because everyone comes to participate rather than watch and listen. A conference should inspire and breed ideas, create valuable connections between people and advance the field. I feel some conferences have lost sight of that.

http://arxivjournal.org/

So far, physicists have been using this approach, but I see no reason why we cannot do the same in Computer Science.

But my initial concern was with improving the conferences themselves. I suppose that intertwined with the purposes of conferences vs. journals. I guess I’d like to see conference proceedings recognized as the low-latency journals they are, and conferences evolve to be more like extended workshops. And ultimately for conferences and conference proceedings (as both exist today) to be decoupled.

I realize that’s a lot to ask of an academic community steeped in traditions, especially when so many people’s careers depend on conforming with those traditions. Senior academics (like Lance Fortnow and Moshe Vardi) will have to lead the way–or industry, which is not held hostage by the status quo, will have to lead by example.

So one question is can something like arxiv.org handle adequate comments that could form reviews? If so, you could implement a comment quality/utility rating and rate reviewers as well as the papers themselves. Reviews can be signed or anonymous; signing will accrue reputation for reviewers.

One concern about open publishing is fair attribution of ideas. If it’s too easy to say “oh, I didn’t see that” but still get credit for the idea, that will discourage people from contributing publications.

Perelman certainly got all due credit with his arxiv-only papers. There was a time when people would have resisted to the idea that publishing an idea on the web was a way to secure credit, but I feel we are beyond this.

@Daniel

2) Open up the review process to allow for additional, non-anonymous input to be considered by the reviewers.

Interesting. It seems that untimely open publishing is somewhat incompatible with anonymous submissions to conferences and journals.

[Cross posted to Daniel Tunkelang's the NoisyChannel]

On point #1… Is this universally true in CS communities? Aren’t the JMLR and MLJ pretty high quality journals in Machine Learning? There are other examples in the AI field. My dissertation in evolutionary computation has loads of journal citations… so I don’t ‘see’ the bad rep of CS Journals.

Agree on #2 and #3. I see myself submitting more to journals for the reasons you raise.

On #5: This is the basic response one should ask of any person advocating to radically change a system that is providing value. Nothing more. Doubt. Ask why and get clear answers and not hand-waving promises of nirvana.

Opening the review process is a bit more contentious because currently authors have expectations of confidentiality in the review process. One concern about open publishing is fair attribution of ideas. If it’s too easy to say “oh, I didn’t see that” but still get credit for the idea, that will discourage people from contributing publications. I suppose one way to ameliorate that is to allow comments on references (or lack thereof) that are delivered integrated into the original paper. This way, it may be possible to correct attribution of ideas after publication.

It’s a messy problem.

1) Pressure conferences to accept work that has been previously published in archives (i.e., make this an exception to the requirement that the work be new / original).

2) Open up the review process to allow for additional, non-anonymous input to be considered by the reviewers.

That seems minimally disruptive to the current process while giving us the benefits of broader dissemination and a more effective review process. It doesn’t resolve some of the larger issues, but at least it would be an improvement.

I wrote about this (half-baked) idea in my blog post.

1) Let the Journals compete for papers with the conference system and attain the proper audience.

Ah! But journals do not compete fairly because there are top-down rules stating that conference papers count more than journal papers.

Many CS researchers will discard journal publications as minor. This is a fact and the ACM article makes this clear: experts decided that journals were less important than conferences in Computer Science. This has been told to all University administrators.

2) In 2009, there is no reason for journals to be any slower than conferences. They used to run slower due to the printing press which we no longer need. If you need to go even faster, I’m afraid you’ll have to forgo peer review and… gulp!… post unreviewed papers on arxiv.

I have had several of my journal articles accepted within 3 months (including revisions). Others took 6 months. Some took longer, but I argue that the journals are mismanaged in such instances.

Of course, my papers were not printed within 3 months, but who cares? Most people download PDFs. If they work with printed copy, they are likely to print their own copies. This is particularly the case in industry.

3) Publishing in a journal is substantially cheaper than publishing in a conference if you include the cost of attending. Thus, journal publications are, if anything, easier for industry researchers. Not having a hard deadline means that legal issues can be dealt with more easily (industry researchers often must get legal approval before submitting a paper).

4) There is no problem with posting papers on arxiv and then having them rejected. If you post a lot of junk on arxiv, then eventually, it will hurt your reputation, but that is your problem. (In time, arxiv could block your uploads too!) If you use arxiv like a responsible researcher, then no ill will come to you. You might argue that “there is too much junk”, yet you are blogging with the rest of us… and there is much junk on the blogosphere.

5) If CS researchers and scientists continue to attend, publish at and found conferences is this not evidence that it is serving a real need?

It almost seems to me that your are advocating conformance: whatever people do right now must be fine because that is what they are doing.

While I’m sure that Vardi and Fortnow have great aims in mind, and I agree with the various points here on the flaws of the conf system…. I can’t buy off on a top down forced reorg into a Journals system that guts the conf system.

Let the Journals compete for papers with the conference system and attain the proper audience. I also have the cynical suspicion that the strong Journal system would result in yet another feudal organizational structure where power is allocated to the in-crowd.

This ‘problem’ seems best solved at the level of individual CS communities. It seems very appropriate for areas like theoretical CS, less appropriate for IR/ML (needs both avenues) and inappropriate for areas like OOPSLA et al.

The ‘solution’ also seems explicitly designed for the benefit of academics seeking to participate in the ‘people ranking’ aspect of academia. In an age of CS where the participation and contribution of industry types to research is at a fantastic level, putting a giant valve in the process of results and idea dissemination seems a step backward.

We already have a hack in place to make it so that not every paper/abstract becomes a talk: poster presentations. But poster presentations are unsatisfying. I think that’s one point where we can improve matters dramatically. I much prefer a slideshare to a poster, for example. Posters are hard to browse, hard to prepare, cumbersome to print and carry…

Meanwhile, talks should be systematically recorded and webcasted. This would immediately mean that there is not other motivation to conference attendance than to network. Indeed, the talks are available to everyone…

These two solutions would change conferences significantly: (1) allow e-talks/slideshares instead of posters (2) webcast talks systematically. For one thing, immediately, we could scale up conferences to as many papers as we want: it would no longer be necessary to limit the number of accepted papers to an arbitrary threshold.

So let’s make a leap and assume that’s not the point of a conference. In that case, it should be about the presentations, the interaction among attendees, or both. Now, how do we optimize for these?

Of course, maybe few people would watch my little video… but if I’m good, people would eventually look forward to my videos/slideshares. If I’m bad, people would ignore me.

I think that technology allows us to increase tenfold our individual broadcasting ability. We have to use that!

im potentially on board with this more shorter presentations. but i dont know what the model of funding for attending conferences is in management for example. i believe the british hci conference this year has taken a near, if not exactly 100% acceptance rate. everything submitted is being shown one way or another, but the best papers are getting larger presentations. the remaining stuff is getting 5mins, 1min, or just taking part in large poster/demo sessions. having said that, some people, who were accepted at the demo level wont be going, because they cant get funding unless they are presenting at a large conference. it kind of throws everything into crazy.

sounds like, tho, a lot of stuff will get shared/disseminated. regardless of if its good.

That said, I don’t object to the incentives you propose, or the idea that a presentation has a life beyond the conference. What I don’t know is what would motivate people who may see the presentations as nothing more than an obligation concomitant with getting papers into the prestigious conference proceedings. Those incentives would have to look good enough on their CVs to matter.

Dividing up conferences does a lot to address the parallel session problem. But it does have a drawback–you’re less likely to meet people outside your narrow area of focus. Of course, the smaller conferences could be co-located and held in parallel with shared breaks and social events, but isn’t that just parallel sessions by another name?

I think talks should be 20 minutes or less, unlike the usual 30 minutes. People waste a lot of time giving needless intro or organizational slides, and forcing them to summarize their paper in 20 minutes or less will make the talks much better and to the point. I don’t see the point of longer talks, because almost 90% of the audience do not read the paper, and can’t grasp the minute details of the idea from the talk. So, conference talks should focus on showing highlights, than explaining the whole thing to an uninitiated audience.

Dividing up large conferences into subject-specific ones is not a bad idea. This is what USENIX has been doing in the last 10 years. Rather than putting everything in USENIX technical, they have FAST, SECURITY, and a bunch of smaller ones. I noticed that this has moved most of the file systems papers to FAST (22 this year) than the Annual Technical one.

Rajib, I’m with you about conferences focusing more on presentations than processings , in case that wasn’t clear from the somewhat loaded questions I suggested. And you raise an interesting point about parallel sessions. But how to solve it? Should the large conferences be partitioned by subject areas (i.e., serial conferences rather than parallel tracks)? Should conferences be more selective? Should presentations be shorter so that they can accept more papers? I assume that making conferences longer is not a realistic option.

In addition, I’d like to add one more suggestion about conferences. I attended both SIGMOD and CIDR this year, and noticed the stark contrasts between them. In CIDR, we had a single track, where in SIGMOD, there were as many as 4 or 5 parallel tracks. Consequently, SIGMOD attendees had to pick and choose which presentation to see. The audience attention was also less (perhaps people were wondering which session to go next?)

But in CIDR, we had a room full of people all focusing on the single track, and had better interaction.

By having multiple parallel sessions, big name conferences are probably saving a lot of money. But that’s also killing off the main reason to attend conferences — meet and interact with people.

I think you nail it when you write: “The peer-review process (and review processes in general) should serve to endorse content–and ideally even to improve it–rather than to filter it.”

Filtering is primarily the purpose of the major Computer Science conferences. They aim to present the “best and brightest” (an 85% rejection rate is not uncommon). This objective is obselete, if it was ever justified.

Myself, I am not all that interested in these authoritative filters. I much prefer to extend my coverage, at the expense of some noise. Afterall, my blogroll contains hundreds of blogs and I cope quite well with the information flow. This comment is not the place to elaborate with my coping methodology: it suffices to say that I am not alone. I mostly stopped watching the TV news. I mostly stopped reading newspapers. I mostly stopped looking forward to the list of accepted papers of major conferences. All these changes in my behavior are strongly related.

I think it is especially important for students to learn to cope and to stop trusting implicitly authoritative filters.