Could rugby teach scientists how to approach the review process?

On the night before the England rugby team attempt to win a first Six Nations grand slam since 2003, it seemed appropriate to have a rugby-themed posting. For those who are not fans of the game with the funny-shaped ball, bear with me…

When a player makes a dive over the tryline and is quickly engulfed in a mass of bodies, it’s naturally hard to tell whether or not the ball has been grounded, and consequently whether a try has been scored. In such cases, it’s usual for the referee to consult with an off-field  fourth official in order to make the decision. This fourth official is sitting in a TV studio and can access video replays of the incident from multiple angles, allowing a more considered appraisal.

Crucially though, the fourth official is not allowed to make the decision – rather, he or she is only permitted to answer the query made by the on-field referee. That query can take one of two forms – “Try: yes or no?” or “Is there any reason a try cannot be awarded?“

There’s a subtle but crucial difference between these requests, and one that reviewers of scientific papers and grants ought to bear in mind. “Try: yes or no” means that the fourth official must decide whether the ball was actually and legally grounded, so in the absence of positive evidence the answer will be no, and no try is awarded. Conversely, “Is there any reason a try cannot be awarded” means that a try can be assumed to have been scored, unless the fourth official finds definitive proof that it hasn’t.

The difference therefore is who gets the benefit of the doubt – the defender or the attacker. Most fans of the sport prefer the latter, as this encourages an attacking game full of ambition, creativity, and risk. Favour the defender too much and you end up with teams adopting a “safety first” conservative approach that may be effective but is often dull.

The parallels with the scientific process are easy to make – the attacker is the applicant, the on-field referee is the editor or funding panel, and the off-field fourth official* is the reviewer. What is the question then that is put to the reviewer, the “scientific fourth official” if you will, by the editor/panel?

I would contend that when reviewing, what should be asked is: “Is there any reason this paper cannot be published?” (or “Is there any reason this grant cannot be funded?”). The default position is acceptance, and the reviewer is there to ensure that there is no flaw that invalidates or terminally weakens the whole submission. If a reviewer instead adopts the mindset of “Should this submission be published/funded: yes or no?”, then they are beginning from a position of rejection and working their way towards acceptance.

As in rugby, these two approaches have significant outcomes in terms of what kind of submissions end up being favoured. Creative, risky ones, or ones that go for safety first. When canvassed, most scientists and funders will profess to favour the creative approach (science is a creative activity, after all); we should ensure that we match deeds with words, and heed the call of the rugby referees – “Is there any reason a try cannot be awarded?”

*Critics of the fourth official say it’s unnecessary and slows the game down. All scientists will concur that the review process is a lengthy and arduous process – although increasingly, there are steps being taken to try and speed it up – but will also concede that it’s important and usually fair.