For the most part, I've been very impressed by the
review process at top architecture conferences (enough to annoy some of my colleagues in other areas :-). But we can always do better. Here are some (hopefully constructive) thoughts:
1. At a PC meeting, if a paper's reviewers can't unanimously agree on a paper, the outcome is often determined by a PC-wide vote. I'm always uncomfortable when the people who read the paper vote 2-1 in favor, but the PC votes 15-16 against the paper. So, essentially, the paper's outcome is decided by one person that never read the paper. I recommend that non-decisive PC votes (margin less than 5?) simply be discarded.
2. A few conferences have experimented with PC meetings that last 1.5 or 2 days. More time does not mean better decisions; it just means that talkative PC members find a way to say the same thing with more words. So let me cast a vote in favor of 1-day PC meetings. Also, a Friday PC meeting allows for better work/life balance -- I assume that's important. I'm curious to see how the ISCA'16 PC meeting plays out: two days, but most PC members will only attend one of the two days.
3. I assume that
Moin will have more to say about the
MICRO'15 review process at MICRO. In a nutshell, about 80+ papers were given 3 weeks to submit a revised paper, in addition to the standard rebuttal. In my opinion, the revision process worked very well. Assuming that a paper is close to being accepted, the authors should want to spend time now to revise the paper than roll the dice at the next conference. I think the MICRO'15 review process hit a sweet spot -- it gets us sufficiently closer to a journal-style process, while retaining the many benefits of conference reviewing.
4. Can we please do away with the rule that a PC member can't have more than 2 accepted papers? Fortunately or unfortunately, I have not been victimized by the rule. But it happens to someone at every PC meeting. It's terribly unfair to the involved grad students. Seemingly, the rule is in place to prevent collusion among PC members. But I just don't buy it. If I was the colluding type, would I stop because there's a cap on my accepted papers? Probably not. I'm ok with a rule that actually stops collusion. I'm not ok with a rule that masks collusion. (FWIW, I have never witnessed collusion at a PC meeting, but I also happen to be relatively naive.)
5. The single most important task for a PC Chair is to find qualified reviewers. The second most important task is to push reviewers towards an active, inclusive, and positive on-line discussion. If these two tasks are done well, the other tasks become a lot less crucial.
6. It has been said many times: as a community, we write harsh reviews. Why? (i) Because we appear smart when we find a flaw. (ii) It's a sign of weakness if we are easily impressed. (iii) Pulling a competitor down is almost equivalent to pulling myself up. Clearly, we all need to dial down the negativity. Write your reviews with the assumption that the submission site will get hacked -- it does wonders for your tone and positivity.
7. It was common practice to reveal author names during the PC meeting. In the past year or so, PC chairs have correctly chosen to not reveal author names at the PC meeting. This is clearly a good practice because it not only eliminates bias at the meeting, but also preserves double-blindness for future submissions of that work. One downside is that it is now much harder to detect incorrectly identified conflicts (both intentional and inadvertent). Entering conflicts for every deadline is both tedious and error-prone. We are well overdue for an automatic conflict management system, perhaps with help from an IEEE/ACM-supported repository of known conflicts.
