The science blogosphere sprang to life when NASA scientists recently reported that they had found a bacterium which, unlike any other known organism, lived off arsenic. Sceptics twittered and critics blogged. Within weeks, excitement about the possible discovery of alien life forms on earth had given way to a critique of NASA and to more prosaic explanations for why its researchers had found arsenic apparently incorporated into bacteria’s DNA. 

It was exactly the kind of lively refutation that we associate with scientific debate at its best. It was a model of post-publication peer review, with microbiologists challenging the research design and probing the NASA scientists’ conclusions. More than that, it was an impressive use of academic blogs and online publication to evaluate the results publicly in something close to real time, while the world was still discussing the research at issue. It was not, as some claimed, proof that the peer review system should be abandoned.

Detractors miss the point

Peer review’s detractors are ever watchful for these kinds of stories and they were quick to seize upon this one. I crossed swords with Richard Smith, former Editor of the British Medical Journal, recently on this question. Richard thinks we should do away with peer reviewing papers before they are published. He asks why we should persist with it when research has been unable to show that it consistently identifies the significance or quality of research papers.¹  

This confuses the system with the principle.  The system covers wildly divergent research fields and produces over a million published papers per year. The principle is that work should be prepared for and subjected to the scrutiny of people who are likely to spot its worth or deficiencies. Peer review doesn’t guarantee work being right, but the required transparency legitimates public questions about whether a research claim has been reviewed.

Human judgement

Peer review’s advocates haven’t been particularly good at setting this out. People are fond instead of criticising peer review. It’s awful, they say, but better than the alternative. This is lazy. It is not awful, it exhibits all the strengths and the weaknesses of a system of organised human judgement.

A human system is flexible and at least capable of spotting good ideas, even if unevenly. It reflects, in any field, the state of that field. It might be stale, narrow, self referential and sloppy. Or it might be dynamic and exacting. Publishing reflects the normal cycles of research – and practices that help when things are on the up, can become part of the problem when they are ossifying. Just like departments and institutes everywhere, some people are still banging a drum, others march on. That is why people go off and create new fields of enquiry with new journals.

Patronage

Instead of saying there is no alternative, we should be clear that there is. It is called patronage. Watch out for its modern packaging, which generally contains reference to new media and cool mates. When Richard tells me that he relies on trusted contacts to alert him to work that deserves attention, it sounds so hip and flexible, not like that stuffy science publishing. Yet if it became the norm it would, in substance, be little different to being accorded merit through the favours of the Medici family.

Peer review - yes, flawed and with all the unevenness of scholarly endeavour - is built upon an aspiration to objectivity and fairness that the public expects from science. Without peer review, some other marker will select papers for us to read. That will be universities with the best-funded public relations departments and principle investigators with the best clubby contacts.

¹ Bornmann, Mutz & Daniel (2010) PLoS One   V5: 12 e14331

[author box]

Tracey Brown is Managing Director of Sense About Science

tbrown@senseaboutscience.org [1]