Advice for writing peer reviews

Rich Sutton

Some advice about writing a review. In my view, the author is king here. The author is the one doing the real work and the success of any meeting or journal depends on attracting good authors. So be respectful to them, while giving your best analysis.

An ideal review goes as follows.

1. The introduction. Summarize the paper is a few sentences. Be neutral, but be sure to include the perspective from which the work might be a good paper. Say what the paper claims to do or show. This section is for the editor and the author. Help the editor understand the paper and show that you as reviewer understand the paper and have some perspective about what makes an acceptable paper.

2. The decision. Give your overall assessment in a few sentences. This includes a clear recommendation for accept/reject. Give the reason for the decision in general terms. e.g., there are flaws in the experimental design which make it impossible to assess the new ideas. or, the authors are not aware of some prior work, and do not extend it in any way. Or, although the experiments are not completely clear, the idea is novel and appealing, and there is some meaningful test of it. Or, the contribution is very minor, plus the presentation is poor, so must recommend rejection. Hopefully you will have many more positive things to say, and will recommend accepting one. The bottom line is: does this paper make a contribution? It should be possible for the editor to read no further than this if he chooses. If there is agreement among the reviewers, this section will be enough for him to write the letter back to the author (or summary review).

3. The argument. Provide the substance that details and review advice.rtf backs up your assessment given in 2. If there are flaws in the experiment, describe them here (not in 2). If there are presentation problems, detail and illustrate them here. In this section you are basically defending your decision in 2. The author and other reviewers are your target audience here. The editor will read this section if there is disagreement among the reviewers.

4. The denouement. Suggestions for improving the paper. It is important that these are suggestions, advice to the author, not reasons for the decision described above. The substance of the review, the decision, is over at this point. Now you are just being helpful. You can make useful suggests whatever the decision was on the paper.

5. The details. I find it useful to save until the end the list of tiny things. Typos, unclear sentences, etc. BTW, if you say they missed some literature, provide a full citation to the work.

If you don't accept a paper, make a clear distinction between changes that would be required for acceptance (for the paper to make an contribution) and which would just make the paper better in your opinion. Authors hate it when a reviewer seems to reject because the paper was not written the reviewer's way.