Categories
General

Are Academic Conferences Broken? Can We Fix Them?

I’d hoped to get through all of the SIGIR 2009 Industry Track before blogging about anything else (such as Yahoo! search going bada-Bing), but clearly I’m taking too long. So I’m following Daniel Lemire’s suggestion that I post a recent comment on Lance Fortnow’s blog (actually a response to his CACM column entitled “Time for Computer Science to Grow Up“) here at The Noisy Channel.

It’s nice to see this piece joining a growing chorus questioning the way we conflate the distinct concerns of disseminating knowledge, establishing professional reputation, and building community. This problem is not unique to computer science, but we are certainly in a position to lead by example in addressing it.

In age where distribution is nearly free, I agree that we should move the filtering role from content publishers to content consumers. There’s no economic reason today why scholarship (or purported scholarship) shouldn’t be published online. Of course, the ability to publish digital content for free (or close to free) does not imply anyone will (or should) read what you write. The blogosphere offers an instructive example: the overwhelming majority of blogs attract few (if any) readers. I suspect that the same holds true for arXiv.org. Of course, peer-reviewed content may not fare that much better, particularly given the proliferation of peer-reviewed venues. Regardless, it makes no sense for publishers to act as filters in an age of nearly-free digital distribution.

That brings us to the question of how researchers should establish their professional reputation–and, in the case of academics, obtain tenure and promotion. Today, they have to publish in peer-reviewed journals and conferences. Even if we accept the weaknesses of the current peer-review regime, we should be able to separate content assessment from distribution. The peer-review process (and review processes in general) should serve to endorse content–and ideally even to improve it–rather than to filter it.

Finally, conferences should primarily serve to build community. I find the main value of conferences and workshops to be face-to-face interaction, and I’ve heard many people express similar sentiments. Part of the problem is that so few presenters at conferences invest in (or have the skills for) delivering strong presentations. But more fundamentally it’s not even clear that the presentations are the point of a conference–after all, an author’s main motive for submitting an article to a conference seems to be getting it into the proceedings.

Here are some questions I’d like to suggest we consider as a community:

What if presentation at a conference were optional, and an author’s decision to present had no effect on inclusion in the proceedings? Would there be significantly fewer presentations? Would those fewer presentation be of higher quality?

What if the process of peer-reviewing conference submissions required the submission of presentation materials rather than (or in addition to) a paper? Would the accepted presentations be of higher quality? Would researchers invest more in presentation skills? What would happen to strong researchers without such skills?

Can we update the traditional conference format to foster more productive interaction among researchers? For example, should we have more poster sessions and fewer paper presentations?

I’d love to see the computer science community take the lead in evolving what increasingly feel like dated procedures for disseminating knowledge, establishing professional reputation, and building community. I’ve tried to do my small part, co-organizing workshops on Human-Computer Interaction and Information Retrieval (HCIR) that emphasize face-to-face interaction and organizing the SIGIR 2009 Industry Track as a series of invited talks and panels from strong presenters. But I’m encouraged to see “establishment” types like Moshe Vardi and Lance Fortnow leading the charge to question the status quo.

By Daniel Tunkelang

High-Class Consultant.

35 replies on “Are Academic Conferences Broken? Can We Fix Them?”

Excellent post.

I think you nail it when you write: “The peer-review process (and review processes in general) should serve to endorse content–and ideally even to improve it–rather than to filter it.”

Filtering is primarily the purpose of the major Computer Science conferences. They aim to present the “best and brightest” (an 85% rejection rate is not uncommon). This objective is obselete, if it was ever justified.

Myself, I am not all that interested in these authoritative filters. I much prefer to extend my coverage, at the expense of some noise. Afterall, my blogroll contains hundreds of blogs and I cope quite well with the information flow. This comment is not the place to elaborate with my coping methodology: it suffices to say that I am not alone. I mostly stopped watching the TV news. I mostly stopped reading newspapers. I mostly stopped looking forward to the list of accepted papers of major conferences. All these changes in my behavior are strongly related.

I think it is especially important for students to learn to cope and to stop trusting implicitly authoritative filters.

Like

I think conferences should focus more on presentations, rather than the proceedings. The noise in conference review process, and the limitation of accepting only a few papers, kill many good papers, and delay publication. Take DB for example: I’ve seen a paper by a colleague get rejected badly at SIGMOD, and then go on to get the best paper award at VLDB. The paper content between the conferences hardly changed, so perhaps it was noise that killed the sigmod submission? Journals are much better than this, and since there are no deadlines, people tend to submit finished work rather than half-baked ideas.

In addition, I’d like to add one more suggestion about conferences. I attended both SIGMOD and CIDR this year, and noticed the stark contrasts between them. In CIDR, we had a single track, where in SIGMOD, there were as many as 4 or 5 parallel tracks. Consequently, SIGMOD attendees had to pick and choose which presentation to see. The audience attention was also less (perhaps people were wondering which session to go next?)

But in CIDR, we had a room full of people all focusing on the single track, and had better interaction.

By having multiple parallel sessions, big name conferences are probably saving a lot of money. But that’s also killing off the main reason to attend conferences — meet and interact with people.

Like

Daniel, thanks for inciting me to post it here. Of course, what goes for students goes even more for tenure / promotion committees. 🙂

Rajib, I’m with you about conferences focusing more on presentations than processings , in case that wasn’t clear from the somewhat loaded questions I suggested. And you raise an interesting point about parallel sessions. But how to solve it? Should the large conferences be partitioned by subject areas (i.e., serial conferences rather than parallel tracks)? Should conferences be more selective? Should presentations be shorter so that they can accept more papers? I assume that making conferences longer is not a realistic option.

Like

By Post Author

But how to solve it? Should the large conferences be partitioned by subject areas (i.e., serial conferences rather than parallel tracks)? Should conferences be more selective? Should presentations be shorter so that they can accept more papers? I assume that making conferences longer is not a realistic option.

I think talks should be 20 minutes or less, unlike the usual 30 minutes. People waste a lot of time giving needless intro or organizational slides, and forcing them to summarize their paper in 20 minutes or less will make the talks much better and to the point. I don’t see the point of longer talks, because almost 90% of the audience do not read the paper, and can’t grasp the minute details of the idea from the talk. So, conference talks should focus on showing highlights, than explaining the whole thing to an uninitiated audience.

Dividing up large conferences into subject-specific ones is not a bad idea. This is what USENIX has been doing in the last 10 years. Rather than putting everything in USENIX technical, they have FAST, SECURITY, and a bunch of smaller ones. I noticed that this has moved most of the file systems papers to FAST (22 this year) than the Annual Technical one.

Like

I’m all for more short presentations vs. fewer long ones. I’d take it even further–I’m a fan of “boaster” sessions that motivate the audience to invest further (by going to a poster session or actually reading a paper).

Dividing up conferences does a lot to address the parallel session problem. But it does have a drawback–you’re less likely to meet people outside your narrow area of focus. Of course, the smaller conferences could be co-located and held in parallel with shared breaks and social events, but isn’t that just parallel sessions by another name?

Like

By Post Author

I think that if conferences remove the requirement to present papers, there will be still a lot of bad presentations – first, people just like to travel and socialize “for free” and, second, they are often just bad presenters and there is no way to prevent them from doing that. There should be some incentives to prepare impressive presentations – like, best presentation awards, when public votes with “dollars” – like it was at CIKM 2009 poster session, or selective videorecording with the follow-up promotion at the official web-site. I actually think that’s a nice idea to allow present not only slides, but also posters for those who has full papers accepted – maybe with some updated info about the paper.

Like

Pavel, you may be right, but I’m not persuaded. First, people could still use travel grants to attend conferences at which they weren’t presenting. Second, the current review process discourages bad writing (or at least considers it as a factor). I see no reason why reviewers couldn’t review slides or a video submission. It might even make for a more efficient review process!

That said, I don’t object to the incentives you propose, or the idea that a presentation has a life beyond the conference. What I don’t know is what would motivate people who may see the presentations as nothing more than an obligation concomitant with getting papers into the prestigious conference proceedings. Those incentives would have to look good enough on their CVs to matter.

Like

By Post Author

from what i know of the academic management conferences, you submit abstracts to them, and people give short presentations. a friend who’s a associate prof in management was completely underwhelmed with my publication record, which i was entirely pleased about at the time. its a community driven event, not for publishing prestige, as you suggest.

im potentially on board with this more shorter presentations. but i dont know what the model of funding for attending conferences is in management for example. i believe the british hci conference this year has taken a near, if not exactly 100% acceptance rate. everything submitted is being shown one way or another, but the best papers are getting larger presentations. the remaining stuff is getting 5mins, 1min, or just taking part in large poster/demo sessions. having said that, some people, who were accepted at the demo level wont be going, because they cant get funding unless they are presenting at a large conference. it kind of throws everything into crazy.

sounds like, tho, a lot of stuff will get shared/disseminated. regardless of if its good.

Like

As a solution, I think we need to use technology a lot more. We are still living in the dark ages as far as conferences are concerned (powerpoint presentations do not count as advanced technology!). For example, I would love to prepare a video that could be presented on the web site of the “conference”. I would prefer to spend a few days preparing a video/slideshare (that would have lasting value), rather than give a talk that only lasts 20 minutes (and with most attendees thinking about the next talk or their own talk).

Of course, maybe few people would watch my little video… but if I’m good, people would eventually look forward to my videos/slideshares. If I’m bad, people would ignore me.

I think that technology allows us to increase tenfold our individual broadcasting ability. We have to use that!

Like

I think the funding question really reflect the value question: why do attendees or their institutions feel the conferences are worth paying for? If the reason is primarily that it’s a prerequisite for those attendees to get their papers published in the conference proceedings, then there’s something very broken, given that we could accomplish all that without holding the actual conference.

So let’s make a leap and assume that’s not the point of a conference. In that case, it should be about the presentations, the interaction among attendees, or both. Now, how do we optimize for these?

Like

By Post Author

Regarding funding… I understand the issue very well, but all this says is that not everyone can attend all conferences all the time. Yet, even Bill Gates cannot attend all conferences. There are non-monetary expenses related to conferences: time, energy, and so on.

We already have a hack in place to make it so that not every paper/abstract becomes a talk: poster presentations. But poster presentations are unsatisfying. I think that’s one point where we can improve matters dramatically. I much prefer a slideshare to a poster, for example. Posters are hard to browse, hard to prepare, cumbersome to print and carry…

Meanwhile, talks should be systematically recorded and webcasted. This would immediately mean that there is not other motivation to conference attendance than to network. Indeed, the talks are available to everyone…

These two solutions would change conferences significantly: (1) allow e-talks/slideshares instead of posters (2) webcast talks systematically. For one thing, immediately, we could scale up conferences to as many papers as we want: it would no longer be necessary to limit the number of accepted papers to an arbitrary threshold.

Like

My two cents was posted below before I saw Daniel’s response here: http://aicoder.blogspot.com/2009/08/response-to-dr-lance-fortnows-cacm.html

While I’m sure that Vardi and Fortnow have great aims in mind, and I agree with the various points here on the flaws of the conf system…. I can’t buy off on a top down forced reorg into a Journals system that guts the conf system.

Let the Journals compete for papers with the conference system and attain the proper audience. I also have the cynical suspicion that the strong Journal system would result in yet another feudal organizational structure where power is allocated to the in-crowd.

This ‘problem’ seems best solved at the level of individual CS communities. It seems very appropriate for areas like theoretical CS, less appropriate for IR/ML (needs both avenues) and inappropriate for areas like OOPSLA et al.

The ‘solution’ also seems explicitly designed for the benefit of academics seeking to participate in the ‘people ranking’ aspect of academia. In an age of CS where the participation and contribution of industry types to research is at a fantastic level, putting a giant valve in the process of results and idea dissemination seems a step backward.

Like

@Neal

1) Let the Journals compete for papers with the conference system and attain the proper audience.

Ah! But journals do not compete fairly because there are top-down rules stating that conference papers count more than journal papers.

Many CS researchers will discard journal publications as minor. This is a fact and the ACM article makes this clear: experts decided that journals were less important than conferences in Computer Science. This has been told to all University administrators.

2) In 2009, there is no reason for journals to be any slower than conferences. They used to run slower due to the printing press which we no longer need. If you need to go even faster, I’m afraid you’ll have to forgo peer review and… gulp!… post unreviewed papers on arxiv.

I have had several of my journal articles accepted within 3 months (including revisions). Others took 6 months. Some took longer, but I argue that the journals are mismanaged in such instances.

Of course, my papers were not printed within 3 months, but who cares? Most people download PDFs. If they work with printed copy, they are likely to print their own copies. This is particularly the case in industry.

3) Publishing in a journal is substantially cheaper than publishing in a conference if you include the cost of attending. Thus, journal publications are, if anything, easier for industry researchers. Not having a hard deadline means that legal issues can be dealt with more easily (industry researchers often must get legal approval before submitting a paper).

4) There is no problem with posting papers on arxiv and then having them rejected. If you post a lot of junk on arxiv, then eventually, it will hurt your reputation, but that is your problem. (In time, arxiv could block your uploads too!) If you use arxiv like a responsible researcher, then no ill will come to you. You might argue that “there is too much junk”, yet you are blogging with the rest of us… and there is much junk on the blogosphere.

5) If CS researchers and scientists continue to attend, publish at and found conferences is this not evidence that it is serving a real need?

It almost seems to me that your are advocating conformance: whatever people do right now must be fine because that is what they are doing.

Like

If we’re revamping the system entirely, I think the following notion is worth discussing:
1. Store all publications (with metadata) in an archive; anyone can add a document as long as rudimentary formatting guidelines are met.
2. have people read/comment/review articles & let others judge the utility of the reviews.
3. Select papers with best reviews for presentation at conferences (with authors’ consent!)

I wrote about this (half-baked) idea in my blog post.

Like

Gene: perhaps a way to get there in incremental steps is:

1) Pressure conferences to accept work that has been previously published in archives (i.e., make this an exception to the requirement that the work be new / original).

2) Open up the review process to allow for additional, non-anonymous input to be considered by the reviewers.

That seems minimally disruptive to the current process while giving us the benefits of broader dissemination and a more effective review process. It doesn’t resolve some of the larger issues, but at least it would be an improvement.

Like

By Post Author

The TOCHI model of allowing articles published in the journal to be presented at the following CHI conference is a step in that direction.

Opening the review process is a bit more contentious because currently authors have expectations of confidentiality in the review process. One concern about open publishing is fair attribution of ideas. If it’s too easy to say “oh, I didn’t see that” but still get credit for the idea, that will discourage people from contributing publications. I suppose one way to ameliorate that is to allow comments on references (or lack thereof) that are delivered integrated into the original paper. This way, it may be possible to correct attribution of ideas after publication.

It’s a messy problem.

Like

@DanielLemire

[Cross posted to Daniel Tunkelang’s the NoisyChannel]

On point #1… Is this universally true in CS communities? Aren’t the JMLR and MLJ pretty high quality journals in Machine Learning? There are other examples in the AI field. My dissertation in evolutionary computation has loads of journal citations… so I don’t ‘see’ the bad rep of CS Journals.

Agree on #2 and #3. I see myself submitting more to journals for the reasons you raise.

On #5: This is the basic response one should ask of any person advocating to radically change a system that is providing value. Nothing more. Doubt. Ask why and get clear answers and not hand-waving promises of nirvana.

Like

@Gene

One concern about open publishing is fair attribution of ideas. If it’s too easy to say “oh, I didn’t see that” but still get credit for the idea, that will discourage people from contributing publications.

Perelman certainly got all due credit with his arxiv-only papers. There was a time when people would have resisted to the idea that publishing an idea on the web was a way to secure credit, but I feel we are beyond this.

@Daniel

2) Open up the review process to allow for additional, non-anonymous input to be considered by the reviewers.

Interesting. It seems that untimely open publishing is somewhat incompatible with anonymous submissions to conferences and journals.

Like

I think it’s by far easier to revamp the journal publication process than the conference process, but changing the journal publication process should make it easier to change the nature of conferences. The key, it seems to me, is to get enough reviewers to achieve critical mass. That makes it attractive to conferences because all they need to do is fight over which well-reviewed paper gets accepted where, whereas the authors can iterate their papers until their reviews converge.

So one question is can something like arxiv.org handle adequate comments that could form reviews? If so, you could implement a comment quality/utility rating and rate reviewers as well as the papers themselves. Reviews can be signed or anonymous; signing will accrue reputation for reviewers.

Like

I agree that migrating journals to an lower-overhead, purely online model is overdue. I do think there are still major questions about how to set up incentives for the requisite money and effort–moving online mitigates but does not eliminate the costs, and does threaten the current revenue model.

But my initial concern was with improving the conferences themselves. I suppose that intertwined with the purposes of conferences vs. journals. I guess I’d like to see conference proceedings recognized as the low-latency journals they are, and conferences evolve to be more like extended workshops. And ultimately for conferences and conference proceedings (as both exist today) to be decoupled.

I realize that’s a lot to ask of an academic community steeped in traditions, especially when so many people’s careers depend on conforming with those traditions. Senior academics (like Lance Fortnow and Moshe Vardi) will have to lead the way–or industry, which is not held hostage by the status quo, will have to lead by example.

Like

By Post Author

I wonder if one good place to start wrt/ arxiv.org or other such sites is to publish workshop proceedings there. Daniel, how about putting all the HCIR stuff in? I wonder if we could get authors’ permissions to put the JCDL 2008 workshop in as well, rather than having it languish on our site.

Like

I think that there are a lot of very “out there” theoretical ideas that would really encourage some creative and exciting thinking if they were presented publicly. If Vannevar Bush had submitted his idea of the web to a conference, it may not have been accepted for a long string of reasons (had he been a nobody).

I’ve been reading Einstein’s biography and he said something along the lines of it not being good for scientists to confine themselves to publishing papers because you ended up writing what you thought would be accepted rather than what is interesting to you. I find myself in agreement.

I submitted a paper about my phd thesis on conversational agents and it was rejected because the reviewers doubted very much that chatbots would be used in business. They were wrong, although I should have written a better and more convincing paper. I do wonder how subjective this reviewing process really is.

I agree with everything Daniel has said so far. I also think using video and the web would be better. I have had papers withdrawn because I couldn’t get to the conference for example. You need funds to travel half way across the world to present a paper. Shame to not publish just because of that, and so many conferences insist on you being there in person.

I feel that conferences need to be more inclusive as well. I know some very talented programmers with very solid and exciting ideas who wouldn’t know how to write a paper for submission. They are not academics and the papers must be of a high standard I agree, but can we make it more accessible? It appears to me that a video presentation, an online demo, an online discussion and these other mediums allow for these non-academics with plenty to offer the field to participate.

I think that peer review is of huge importance to us all, but at the same time rejecting 85% of papers is quite a lot when you think about it. I want to read all the rejections, and like Daniel I have no trouble dealing with the extra noise.

This is impractical but I’ll suggest it anyway:

Could we send through a proposal for a presentation, like an abstract for example and then as an organiser you have an idea of the types of projects and can form natural categories for this year. Duplicates can be coupled together an the authors notified and it can be suggested that they communicate together and maybe do a joint presentation of some kind (rather than reject 9 and accept 1 of them). Then once organised in groups or alone, people upload their presentations 2 weeks before the conference to give others time to view them.

The conference is all about discussion rather than presenting this way.

Wouldn’t you value a person who has answered an hour of pertinent questions from his/her peers rather than one that has talked at/to them for 30mins? More importantly, someone who during a discussion has been able to give more to the field?

Conferences become about everyone, because everyone comes to participate rather than watch and listen. A conference should inspire and breed ideas, create valuable connections between people and advance the field. I feel some conferences have lost sight of that.

Like

CJ, not sure which Daniel you’re agreeing with (me or Daniel Lemire), but I like what you are suggesting. I was thinking a similarly impractical thought: let presenters assume that the audience has read the paper, and use the entire time for Q&A. After all, there’s no point in wasting the scare resource of live access. Presumably this would also allow “presentation” slots to be much shorter.

Like

By Post Author

lol, sorry! I meant Daniel Lemire – I always say his name in French in my head and yours in English so there is a clear distinction in my mind lol

Like

The system is broken.

I complained to a conference organizer that there were too many routine and low-quality papers by very junior people. The answer: “Well, we have to give our graduate students someplace to publish so they can get jobs.”

I complained to the senior co-author of a paper presented in unintelligible English by his student that the presentation was terrible (even if the content seemed like it might be good). The answer: “He needs an opportunity to practice.”

Of course junior people need to publish and to polish their presentation skills. But when that becomes the *primary* purpose, trumping the usefulness of the presentations to the audience, something is very wrong.

Like

I agree — we have to distinguish publishing from presenting. And there’s no reason that the expenses associated with organizing conference should be necessary for people to prove the worthiness of their research.

Like

By Post Author

Comments are closed.