Might waiving anonymity actually improve the fairness of peer review?

Earlier this year, Bravo et al. published a fascinating study looking at the effect of open peer review on referee behaviour. The results hint at a potentially ugly truth: anonymous peer review is biased towards negativity.

Data for the paper were drawn from an open review pilot conducted in five Elsevier journals (two Open Access, three subscription-based) from 2014-2017. Referees were asked in advance for permission to publish their reviews, and additionally given the option of keeping or waiving their anonymity.  

The conclusions make good reading for the proponents of open review: Bravo et al. concluded that (1) open peer review does not appear to compromise the rigour of peer review, and (2) willingness to review, turnaround time, language, and recommendations are all (yes, naysayers, that’s ALL) unaffected when reviews are made public instead of kept private.

There was one big difference, however. When referees waived their anonymity, they were much, much more likely to view the manuscript favourably. Rejection decisions dropped from 30% down to 7%, major revision rates stayed roughly the same at around 28% or so, requests for minor changes only leapt up from 28% to over 40%, and decisions to accept went from 10% to over 20% (all quoted values approximate, and based on squinting at the figure after consumption of a large glass of wine; see Figure 4 in the paper for exact measured proportions).

The authors’ interpretation of this effect is interesting, and worth quoting verbatim:

“…the distribution of recommendations by referees who accepted to have their names published with the report was noticeably different, with many more-positive recommendations. Given that revealing identity was a decision made by referees themselves after completing their review, it is probable that these differences in recommendations could reflect a self-selection process. Referees who wrote more-positive reviews were more keen to reveal their identity later as a reputational signal to authors and the community. However it is worth noting that only a small minority of referees (about 8.1%) accepted to have their names published together with the report.”

In other words, these changes are interpreted either as cowardice (chickens!) on the part of anonymous referees – with people being afraid of retaliation from vindictive colleagues – or toadying (arselickers!) on the part of non-anonymous ones – with people wanting to let authors know that they did them a favour so that perhaps they owe them one in return. 

Neither interpretation is edifying. It paints a bleak picture of both sides of the reviewer equation, regardless of which angle you favour. Let’s not misunderstand one another here: unfair peer review is unscientific, retaliation against tough review(er)s is unscientific, and tit-for-tat venality is unscientific. 

While Bravo et al. may well be right that self-selection is responsible for the higher frequency of “accept” decisions in the non-anonymous camp, the reciprocal loss/gain of the reject/minor changes categories might well merit further consideration. 

Importantly, the proposed interpretation also assumes that the decision to waive anonymity was made after completing the review, rather than before starting. In other words, it assumes that referees who write positive reviews gladly waive their anonymity, rather than non-anonymous referees actually being more favourable in general.

This begs a very important question. Is the more-positive output from non-anonymous referees in fact an indictment of anonymous peer review? Are the more favourable reactions –  evidenced by higher rates of only “minor changes” requested – actually a better reflection of the science? Put more controversially, is anonymous peer review intrinsically biased to be more hostile? 

Possible reasons for this are easy to imagine. The uglier (and unscientific) ones are basically a manifestation of road rage – people taking advantage of the anonymity provided by peer review to vent frustrations, settle scores, and handicap perceived competitors or structurally equivalent peers. All things that should be set aside when sitting down to review a paper but often are not. 

Equally, it’s worth stressing that a seemingly hostile review needn’t be motivated by bitterness or conspiracy – it can just as easily be produced by thoughtlessness, itself often prompted by a lack of sufficient time. Not carefully considering the effect that your words will have on readers can all too easily result in an offhand or poorly-composed comment having a devastating impact at either editorial or authorial level. The bottom line is that the impunity offered by anonymity is likely to promote – or, at the very least, license – behaviours that people would not do otherwise. 

Writing a non-anonymous review forces you to think about your language very carefully, because everybody knows the trauma associated with publication, and nobody wants to get casually squashed. You are far, far less likely to submit a review in a rush if you know the authors will see who you are.

Human beings have an exquisitely fine-tuned sense of fairness, and what enrages many about peer review is the arbitrary unfairness of the judgements. Some of these perceived slights can fester over years, poisoning the chances of a productive and open discussion, and thereby making science itself the loser. Pink Floyd nailed that sentiment in “Lost for Words” (So I opened the door to my enemies/and asked could we wipe the slate clean?/and they tell me to please go and fuck myself/no, you just can’t win). 

Conversely, and more optimistically, it’s genuinely remarkable how much more mature, civilised, and – yes, believe it! – cordial peer review can be when it’s not anonymous. Waiving anonymity during review has the perverse effect of making judgements more impersonal, because it shows that the reviewer is focusing on the science and not using anonymity as a cloak to take shots at the authors from behind cover. 

Peer review was never intended to be trial by ordeal, or testing to destruction. It is supposed to be firm, fair, unbiased, constructively critical, and helpful. It cannot be infallible. A problem on the receiving end is that the currency of publication (driven largely by hypercompeittion for funding resources) has risen so high that it’s all too easy to see a tough review as being part of a faceless conspiracy against your own career prospects.

As scientists, and ultimately in service to science itself, we should encourage referees to waive anonymity, should encourage them to do so upon beginning a review and not upon completing it, and (if need be) enforce penalties against people who engage in retaliatory and unscientific behaviour against peers who are providing an objective service. Try it! And let us know how it goes.