||Reinforcement Learning and
edited by Rich Sutton
The ambition of this
page is to collect wisdom on writing reviews of scientific papers which
provide help for the authors and the chairs.
First, remember that the author is the one doing the real work, and
that the success of any meeting or journal depends on attracting good
authors. So be respectful to them, while giving your best analysis.
Start from a high level - what is the problem, how does the paper
attempt to contribute to this problem, and does it succeed.
The details of an ideal review follow:
- The introduction. Summarize the paper is a few sentences. Be
neutral, but be sure to include the perspective from which the work
might be a good paper. Say what the paper claims to do or show. This
section is for the editor and the author. Help the editor understand
the paper and show that you as reviewer understand the paper and have
some perspective about what makes an acceptable paper.
- The decision. Give your overall assessment in a few sentences.
This includes a clear recommendation for accept/reject. Give the reason
for the decision in general terms. e.g., there are flaws in the
experimental design which make it impossible to assess the new ideas.
or, the authors are not aware of some prior work, and do not extend it
in any way. Or, although the experiments are not completely clear, the
idea is novel and appealing, and there is some meaningful test of it.
Or, the contribution is very minor, plus the presentation is poor, so
must recommend rejection. Hopefully you will have many more positive
things to say, and will recommend accepting one. The bottom line is:
does this paper make a contribution? It should be possible for the
editor to read no further than this if he chooses. If there is
agreement among the reviewers, this section will be enough for him to
write the letter back to the author (or summary review).
- The argument. Provide the substance that details and backs up
your assessment given in 2. If there are flaws in the experiment,
describe them here (not in 2). If there are presentation problems,
detail and illustrate them here. In this section you are basically
defending your decision in 2. The author and other reviewers are your
target audience here. The editor will read this section if there is
disagreement among the reviewers.
- The denouement. Suggestions for improving the paper. It is
important that these are suggestions, advice to the author, not reasons
for the decision described above. The substance of the review, the
decision, is over at this point. Now you are just being helpful. You
can make useful suggests whatever the decision was on the paper.
- The details. I find it useful to save until the end the list of
tiny things. Typos, unclear sentences, etc.
BTW, if you say they missed some literature, provide a full citation
to the work.
If you don't accept a paper, make a clear distinction between
changes that would be required for acceptance (for the paper to make an
contribution) and which would just make the paper better in your
opinion. Authors hate it when a reviewer seems to reject because the
paper was not written the reviewer's way.