Tips from a journal editor: being a good reviewer

Peer review is one of the foundations of science. To have research scrutinized, criticized, and evaluated by other experts in the field helps to make sure that a study is well-designed, appropriately analyzed, and well-documented. It helps to make sure that other scholars can readily understand, appreciate, and build upon that work.

Of course, peer review is not perfect. Flawed studies are published, and peer reviewers may miss critical problems or errors in particular studies. Reviewers often do not have the time, nor the inclination, to dig deeply into a study’s methods, or assumptions, or supporting materials, in order to find errors or flaws in a research paper.

Even though peer review is not perfect, as a journal editor I rely heavily on the evaluations and advice provided by peer reviewers. We spend a great deal of time trying to find the right reviewers for each and every paper that we put through the peer review process, and an equally large amount of time reading, evaluating, and putting into appropriate context the responses that we receive from our reviewers.

What is ironic, however, is that despite the importance of peer review in science, it’s not a skill that we typically directly address in our graduate programs, nor in our professional societies. I’m not aware of graduate seminars in “how to be a good peer reviewer”, nor are there materials easily available for scholars to reference when they are asked to undertake a particular peer review task for a journal.

So I thought I’d provide some guidance for journal peer reviewers, at least from the perspective of one of the editors of Political Analysis. Here’s what makes for a good reviewer:

• Evaluate the quality of the research reported in the paper you have been asked to review.
• Refrain from judging whether the paper is suitable for the journal (unless you are asked for that advice).
• Avoid providing a lengthy list of typographical errors.
• Reveal your conflicts immediately.
• Be timely.
• Be brief.

Now, I’ll elaborate a bit more on each point.

First, as a reviewer, your job is to give me an evaluation of the quality of the research reported in the paper that you are provided. Does the theoretical model or hypothesized expectations make sense? Is the data appropriate for testing the hypotheses presented? Are the assumptions of the methodology sound? If the author develops a new theoretical model or estimator, is the math correct? Do the results presented make sense? Are the conclusions appropriate, given the analysis presented in the paper? Those are the most important questions that reviewers need to consider. As a journal editor, I need to know whether a scientific peer reviewer believes that the research reported in the paper is sound.

Second, unless I ask you specifically for your opinion about whether the paper might be suitable for our journal, that’s not advice I’m looking for. After all, I’ve just asked you to review the paper for the journal, so clearly I think that it’s appropriate for peer review! It’s our decision as editors whether a paper might or might not be appropriate for the journal, and while you might have opinions on the matter, in general that’s not what I’m looking for. So refrain from writing in your review that the paper should not be published in our journal because you don’t think the content is suitable for the journal. Unless I ask for that advice, I’m not looking for it.

Third, minor comments — typographical errors in the paper, missing citations, formatting issues, the writing style — really aren’t important to discuss in your review, unless they are so egregious that they detract from your ability to understand what is going on in the paper. Sure, if you want to give the author feedback about these sorts of minor issues, that’s fine, but assuming that we proceed and allow an author to revise their paper, these will be dealt with in the revision, copyediting, and production process. They don’t play an important role in the initial stages of review, unless these problems truly make a paper nearly impossible to understand.

Fourth, ethics. Our journal reviews most papers currently using a double-blind peer review process. That means the editors know who the authors are, and we of course know who the reviewers are. We try our best to avoid asking colleagues to review a paper if they have obvious conflicts of interest. For example, we try to avoid asking for a review from a scholar who we know is a collaborator with an author, who is at the same institution as the author, or who may have some other intellectual conflict with the author. So if you get a paper to review, and you believe you have a conflict of interest that the editor is not aware of, please let us know, as sometimes we may not know of these conflicts. Disclosing the conflict is important; editors can then judge the merits of the review, and proceed accordingly.

Next, be timely. No one wants the review process to drag out indefinitely. Don’t accept review invitations if you cannot fulfill the request in a timely manner. If something comes up that will significantly delay your ability to provide the review in a timely manner, let the editor know. We don’t want to unnecessarily burden you with review requests, and we certainly don’t like to send all of those nagging emails. Being a good reviewer means being providing a timely review.

Finally, be brief and focus on the most important criticisms that you have of the paper. For most manuscripts, we don’t need an evaluation that stretches on for more than four to six paragraphs; in many situations, a review can be far less lengthy than that. We certainly don’t need reviews that go for pages and pages; those tend to be less useful for editors in their decision making, and less helpful for the authors. Stay focused. Let the editors know what you think are the central contributions and important problems with the paper.

While there are no cut-and-dried standards for the conduct of peer review, many journals are starting to provide guidelines for reviewer expectations, in particular when it comes to ethics. Political Analysis has some guidance for reviewers available on our journal’s website, and we will continue to update those guidelines. Hopefully other journals, and our professional societies, will do more in the future to provide guidance to scholars about peer review.

So your job as a peer reviewer for a journal is important. As an editor, we need your evaluation of the scientific merit of the paper we ask you to review. Being a good reviewer is not easy, but it is an important part of being a member of the scientific community.

Featured image: Typing. CC0 via Pixabay.