I don’t know anybody who likes peer review. The complaints are many but boil down to two main concerns: peer review is way too slow and the quality of the comments varies far too widely.
I want to focus on the second issue. Part of the problem, I think, is that editors provide too little direction to peer reviewers and peer reviewers seem to have far too much discretion to assess the manuscript in whatever way they wish. The result, in my experience, are reviewers that frequently want you to write a completely different paper or book, hammer your choice of methodological tool or theory (based on personal preference), or provide criticisms that clearly indicate that they did not read the manuscript carefully enough (to be fair, sometimes this criticism is a signal that the author needs to make things clearer or more pronounced). Continue reading
I think the days of “free for all” reviews needs to end. Instead, editors should consider adopting a list of questions and holding reviewers to answering ONLY those questions (no more additional comments or recommendations!).
Here’s what my reviewer form would look like:
1) Is the argument presented in the paper internally consistent? If not, please identify inconsistencies in the argument.
2) Does the paper make an original contribution to the literature? What is that contribution and what is its magnitude (on a scale of 0-10, with zero being none and 10 being ground-breaking)?
3) Does the evidence presented adequately support the arguments presented in the paper? If not, identify weaknesses or areas where additional evidence would be helpful.
4) Are there any plausible alternative explanations/arguments, given the evidence presented in the paper, that the author should consider seriously?
I wouldn’t ask reviewers to recommend publication or not. I would simply limit them to answering these four questions and make a decision based on my own reading of the manuscript and these reviews.
Why these four questions? I think peer review should be about assessing whether the manuscript makes any type of contribution (big or small) to the literature and whether the paper is sound in terms of scholarly rigour. Contribution is important (e.g. question 2 above) since higher ranked, general political science journals, will probably emphasize larger contributions, but that should be only part of the calculation (many small contributions are just as important as one or two major ones!). Limiting reviewers to rigour is also important because far too often, individual reviewer preferences about research topics and questions, approaches, methods, theories, and political leanings, seem to take precedence when they shouldn’t. If I choose to do a descriptive, analytical paper, that shouldn’t automatically lead a reviewer to reject a paper just because they wish I wrote something different (normative or explanatory). Reviewers instead should be assessing questions 1, 3 and 4.
What do you think? Would you add anything else to my reviewer form? Would this form and procedure generate different outcomes?