In the review process for top conferences, conventional wisdom says that rebuttals don't usually change the mind of the reviewer... they only help the authors feel better. The latter is certainly true: after writing a rebuttal, I always feel like I have a plan to sculpt the paper into great shape.
Here's some data from a recent PC meeting about whether rebuttals help. For the MICRO 2011 review process, reviewers entered an initial score for overall merit and a final score after reading rebuttals and other reviews. While many factors (for example, new information at the PC meeting) may influence a difference in the two scores, I'll assume that the rebuttal is the dominant factor. I was able to confirm this for the 16 papers that I reviewed. Of the roughly 80 reviews for those papers, 14 had different initial and final scores (there were an equal number of upward and downward corrections). In 11 of the 14 cases, the change in score was prompted by the quality of the rebuttal. In the other 3 cases, one of the reviewers convinced the others that the paper had merit.
For 25 randomly selected papers that were accepted (roughly 125 reviews), a total of 14 reviews had different initial and final scores. In 10 of the 14 cases, the final score was higher than the initial score.
For 25 papers that were discussed but rejected, a total of 19 reviews had different initial and final scores. In 14 of the 19 cases, the final score was lower than the initial score.
It appears that rebuttals do play a non-trivial role in a paper's outcome, or at least a larger role than I would have expected.
a lot of score movement happens as a result of discussions. Are your stats post-rebuttal and pre-discussion, or overall ?
ReplyDeleteI checked the scores after the meeting. I agree that a few corrections may have happened at the meeting, so I should have taken a snapshot right before. I think in-meeting score adjustments only happened in the rare event that a paper was being turned down with universally positive scores. For my batch of 16 papers, the 11 score corrections that I attributed to the rebuttal happened prior to the meeting.
ReplyDeletehmmm...more information is good...on a unrelated "process" note, shouldn't such info either remain confidential or be disseminated through a "better" conference broadcast system or official means?
ReplyDeleteI don't see why such information should be confidential. As a community, we should be curious about whether our process works. It would be great if some future PC chair analyzes the data thoroughly and includes it in his/her message.
ReplyDeleteOn another related note, I have no idea why our overall merit scores are not revealed during the rebuttal phase. Authors know that their scores change sometimes; it is a sign that the process works; the PC should not be ashamed of admitting that they changed their mind.
The overall scores changed - but what percentage of them changed the overall outcome of the paper from reject-> accept or vice versa? That is the metric that really matters for submissions, not just absolute change in the score.
ReplyDeleteDave: Agreed. But... the dynamics of a PC meeting are very hard to predict. Papers get accepted and rejected for a combination of several reasons: a convincing champion, a convincing anti-champion, an innocent remark by someone that never read the paper, the PC being bored by the Nth paper on topic X, etc. Attributing the outcome to something in the rebuttal is next to futile.
ReplyDeleteGood point - take home message must be, for the minor effort it takes to write a good rebuttal (even if you're just enumerating the things you'll fix in the re-submission) it can't hurt your chances!
ReplyDelete