Bad Advice

Five years ago, our group had visited Micron for informal research discussions (documented in an earlier blog post).  It is fair to say that we were generally discouraged from messing with memory chips, and instead encouraged to look at innovations outside the memory chip.  We were advised (warned) that even introducing a 0.5% area overhead to a memory chip would be untenable.  And this was the dominant view within the memory industry.  Many academic reviewers over the years have provided similar advice.

As is now evident, that advice was mis-guided.

Just as there are markets for a variety of processor chips, there also are emerging markets for a variety of memory devices.  Some memory devices will incur a cost penalty in order to provide lower latency or lower energy.  For example, see the slide below that was presented by Hynix at an ISCA 2016 keynote, arguing for a memory chip that offers 30% lower latency, while incurring a 28% area overhead.

From the ISCA 2016 keynote talk by Seok-Hee Lee (SK Hynix)

Micron's Hybrid Memory Cube is also designed with many small DRAM arrays that reduce "overfetch" but occupy more area.  This was the key idea in our ISCA 2010 paper -- an idea that received a lot of criticism from reviewers (at least one "Strong Reject"), ISCA attendees (at least one nasty question during the Q&A), and designers at Micron (the ominous reality check on area overheads).

Hopefully, program committees will no longer cite the "cost-sensitive memory industry" to reject otherwise good papers.  Market forces are constantly evolving -- they should ideally not come in the way of a strong technical argument.

Random Thoughts on the Review Process

For the most part, I've been very impressed by the review process at top architecture conferences (enough to annoy some of my colleagues in other areas :-).  But we can always do better.  Here are some (hopefully constructive) thoughts:

1. At a PC meeting, if a paper's reviewers can't unanimously agree on a paper, the outcome is often determined by a PC-wide vote.  I'm always uncomfortable when the people who read the paper vote 2-1 in favor, but the PC votes 15-16 against the paper.  So, essentially, the paper's outcome is decided by one person that never read the paper.  I recommend that non-decisive PC votes (margin less than 5?) simply be discarded.

2. A few conferences have experimented with PC meetings that last 1.5 or 2 days.  More time does not mean better decisions; it just means that talkative PC members find a way to say the same thing with more words.  So let me cast a vote in favor of 1-day PC meetings.  Also, a Friday PC meeting allows for better work/life balance -- I assume that's important.  I'm curious to see how the ISCA'16 PC meeting plays out: two days, but most PC members will only attend one of the two days.

3. I assume that Moin will have more to say about the MICRO'15 review process at MICRO.  In a nutshell, about 80+ papers were given 3 weeks to submit a revised paper, in addition to the standard rebuttal.  In my opinion, the revision process worked very well.  Assuming that a paper is close to being accepted, the authors should want to spend time now to revise the paper than roll the dice at the next conference.  I think the MICRO'15 review process hit a sweet spot -- it gets us sufficiently closer to a journal-style process, while retaining the many benefits of conference reviewing.

4. Can we please do away with the rule that a PC member can't have more than 2 accepted papers?  Fortunately or unfortunately, I have not been victimized by the rule.  But it happens to someone at every PC meeting.  It's terribly unfair to the involved grad students.  Seemingly, the rule is in place to prevent collusion among PC members.  But I just don't buy it.  If I was the colluding type, would I stop because there's a cap on my accepted papers?  Probably not.  I'm ok with a rule that actually stops collusion.  I'm not ok with a rule that masks collusion.  (FWIW, I have never witnessed collusion at a PC meeting, but I also happen to be relatively naive.)

5. The single most important task for a PC Chair is to find qualified reviewers.  The second most important task is to push reviewers towards an active, inclusive, and positive on-line discussion.  If these two tasks are done well, the other tasks become a lot less crucial.

6. It has been said many times: as a community, we write harsh reviews.  Why?  (i) Because we appear smart when we find a flaw. (ii) It's a sign of weakness if we are easily impressed. (iii) Pulling a competitor down is almost equivalent to pulling myself up.  Clearly, we all need to dial down the negativity.  Write your reviews with the assumption that the submission site will get hacked -- it does wonders for your tone and positivity.

7. It was common practice to reveal author names during the PC meeting.  In the past year or so, PC chairs have correctly chosen to not reveal author names at the PC meeting.  This is clearly a good practice because it not only eliminates bias at the meeting, but also preserves double-blindness for future submissions of that work.  One downside is that it is now much harder to detect incorrectly identified conflicts (both intentional and inadvertent).  Entering conflicts for every deadline is both tedious and error-prone.  We are well overdue for an automatic conflict management system, perhaps with help from an IEEE/ACM-supported repository of known conflicts.

Rebuttals and Reviews

In the review process for top conferences, conventional wisdom says that rebuttals don't usually change the mind of the reviewer... they only help the authors feel better.  The latter is certainly true: after writing a rebuttal, I always feel like I have a plan to sculpt the paper into great shape.

Here's some data from a recent PC meeting about whether rebuttals help.  For the MICRO 2011 review process, reviewers entered an initial score for overall merit and a final score after reading rebuttals and other reviews.  While many factors (for example, new information at the PC meeting) may influence a difference in the two scores, I'll assume that the rebuttal is the dominant factor.  I was able to confirm this for the 16 papers that I reviewed.  Of the roughly 80 reviews for those papers, 14 had different initial and final scores (there were an equal number of upward and downward corrections).  In 11 of the 14 cases, the change in score was prompted by the quality of the rebuttal.  In the other 3 cases, one of the reviewers convinced the others that the paper had merit.

For 25 randomly selected papers that were accepted (roughly 125 reviews), a total of 14 reviews had different initial and final scores.  In 10 of the 14 cases, the final score was higher than the initial score.

For 25 papers that were discussed but rejected, a total of 19 reviews had different initial and final scores.  In 14 of the 19 cases, the final score was lower than the initial score.

It appears that rebuttals do play a non-trivial role in a paper's outcome, or at least a larger role than I would have expected.