Revisions required

The hamsters first shook my faith in peer review. They were the models in a study investigating whether Cr(vi) ions are released from chromium alloys implanted in the body. However, it was obvious to me – even as a first year PhD student with a rudimentary knowledge of electrochemistry – that the conclusion of the paper was overblown; a flaw in the experimental conditions meant the results of the accelerated corrosion tests didn’t translate to real-world conditions. I still wonder whether the hamsters’ sacrifice was worth it.

The paper was published in a respected, peer-reviewed journal – albeit one that didn’t specialise in corrosion. It’s impossible to know what happened in the review process, but I suspect there just wasn’t a corrosion scientist on the panel of peer reviewers – a simple case of missing expertise. (The fact that no one else appeared to have criticised the paper in the 25 years between its publication and my reading of it strengthens my suspicions that it never reached a corrosion audience.)

It’s not surprising that oversights like this happen occasionally. It’s difficult to find at least two independent experts whose collective knowledge covers every single aspect of a paper, particularly as research projects become more interdisciplinary. That’s not to say journals don’t try. I used to help coordinate the peer review of opinion-based articles for a biology journal. In training, the first thing drilled into me was the importance of taking the time to acquire a diverse set of reviewers for a paper. That didn’t mean just scientific expertise; the ideal reviewing panel would contain researchers of a mix of genders, geographical locations and career stages.

Finding such a group could prove tricky, particularly when you’re also asking reviewers to provide a thoughtful, detailed response within three weeks, all for free. But it was worth the extra effort. Not only does such diversity help to counter any unconscious biases held by individual reviewers, but the different backgrounds and viewpoints held by each expert would often produce a richer and more thorough assessment of the paper.

Peer review may be imperfect, but so is any system of qualitative assessment. The problem, as explored by Rachel Brazil in her investigation of alternative review systems, is that its flaws are magnified because academic careers now strongly depend on peer review outcomes. However, we shouldn’t overlook the fact that peer review works well in many cases, and the feedback it provides benefits both scientists and scientific progress. Further experiments are needed to improve the reliability, equity and fairness of this feedback; peer review requires revision, not rejection.