Artwork by Oliver Hoeller.
In the latest “How to…” guide, TIR offers a few tips and recommendations on the critical evaluation of scientific papers.
Reading scientific papers is a vital part of every researcher’s regular activity. It’s a means of keeping up with the latest developments in your field, learning about new discoveries elsewhere, and getting to grips with novel methodologies and assays.
However, a key element of this activity is the critical assessment of the material being consumed. It’s important to apply the same rigour, scepticism, and discipline to reading others’ work as you would to analysis of your own data. A paper should never, ever, be taken at face value, and given the growing popularity of preprints and the possibility of significant change in the way scientific data is disseminated, it’s more important than ever that young scientists are capable of objective scrutiny of outside work.
Critical assessment of a paper doesn’t end when it’s published; arguably, it never ends. This is how you can contribute to that process.
Order of reading:
The best way to critically read a paper is:
- Figures + Figure legends;
- Introduction (plus References)
- Results (plus Figures/Figure legends)
- Materials & Methods.
The key thing about this approach is that it places the data first, before the authors have a chance to justify/explain themselves. A well-constructed paper should be intelligible from the Figures and Figure legends alone, and this is often a useful source of first impressions. A classic shortcoming of published scientific work is a disconnect between the data (in the Figures), and the interpretation of that data (in the text). By going over the Figures first (using only the Abstract to guide your sense of what the overall story is about), you will be able to draw your own conclusions and get a sense of the overall quality before you see what interpretation the authors are favouring.
Obviously this approach is far more time-consuming that just reading the whole paper in publication order (Abstract, Introduction, Results+Figures, Discussion, Materials & Methods), and may not be practical if you are simply reading a paper for interest.
If, however, a proper appreciation of the paper is important (for example, for a Journal Club), then the “Figures first” approach is definitely recommended. If possible, it’s also worth spreading out the process over several days, so that you can keep returning to it with a fresh mind. If you have the time, then allocating 3-4 days (not full time!) is good.
This would mean something like:
- Day 1: skim the Abstract, Figures, Figure legends to get a flavour of the story.
- Day 2: go over the Abstract, Figures, Figure legends in detail.
- Day 3: quickly skim the Abstract, Figures, Figure legends again to refresh your memory, then do the other sections in detail. Note the structure of the argument as you go along.
- Day 4: skim over the whole thing again, and then write your report.
The key advantage of spreading the task over several days is that it helps to take the emotion out of the process. If there’s something that’s annoying in the data presentation or writing, you have time to calm down before you put the report together. Equally, there’s time to think over the authors’ argument carefully and decide what you agree and don’t agree with.
In chronological order:
Part 1: Abstract
Read this first to give you a sense of the subject area, organism, and methodological approach used in the paper. At this stage, it’s just a rough guide so you can put the data in context; you will read it again after you’ve finished going through the data.
Part 2: Figures + figure legends
Go through the figure legend for each figure LINE BY LINE.
– Does the title of the figure legend explain the main question addressed in the figure?
For each panel/assay:
– Have the authors explained what assay is being used?
– Have they provided a positive and a negative control?
– Do the choices of positive and negative control seem appropriate?
– Have they quantified their data?
– Is the quantified data better than presenting all of the raw data? (this is the null hypothesis for quantification – if the quantified data are harder to follow or less informative than the raw data would have been, then something is amiss. Example: presenting a mean and an error bar when only two independent experiments have been carried out)
– Have they explained their measure of reproducibility? (technical replicates, biological replicates, independent experiments – are YOU sure of the difference between these things? See TIR’s guide if not.)
– Do you agree with their reproducibility measures, or do you think the authors are unclear on the difference between technical/biological replicates and independent experiments?
– If they have carried out significance tests, do those tests seem justified? (for low sample numbers, significance tests are unlikely to be meaningful)
– Are there scale bars on their micrographs?
– In blots, have they provided an input sample? Is the % value of the input disclosed? Are the % values of the other samples disclosed? (a high number of published fractionation/immunoprecipitation/pulldown figures are very vague on these points)
– How much of the data presented relies on “representative examples”? Ideally, a representative example should be accompanied by quantification of a larger sample.
– Conversely, when quantification of large samples has been carried out, have examples of exactly what was quantified been provided? (Otherwise, “garbage in, garbage out” may apply)
– (Plus point) Have they provided schematics or similar to explain what they did for non-specialists?
Once you’ve gone through the figures and drawn your own conclusions, you can now see what story the authors have constructed around the same data and if their interpretations are consistent with yours…
Part 3: Abstract (take two)
– Does the abstract explain what the main question is, and what the main finding is?
– Do you have a rough sense of the subject area, organism, and technical approaches?
Part 4: Introduction + References:
(Generally, the justification for the research can be skimmed over – this is often just framing)
Tip: copy and paste the references list into a new document and examine that alongside the manuscript text. This will allow you to check the references on the go.
– (For specialists) Have they cited the right papers for each particular claim? Can YOU think of any relevant articles that are missing?
– Is there an overreliance on citation of reviews? Papers should cite other primary literature whenever possible.
– Have the authors explained the background of the field? What is the big question or subject area?
– Have the authors defined what the extent of current knowledge is?
– Have the authors identified where there is a knowledge gap, which they will attempt to address?
– (optional) Have they briefly outlined how they attempted to address it?
– (optional) Have they provided the main conclusion in advance? (rhetorical device – helps prepare the reader to accept what they say)
Part 5: Results
*** Cross-check your assessment of the results with your impressions from the figures+figure legends alone***
For each figure:
– Have they explained the rationale behind the experiment(s)?
– Have they explained what their positive and negative controls were?
– Are the positive and negative controls appropriate? (negative controls are sometimes misleading)
– What controls would YOU use if you were doing this experiment?
– Have the authors quantified the data?
– Is it possible/easy to relate the quantified data to the raw data?
– How would YOU have quantified the data? Can you think of a better presentation? (e.g. bar vs line charts, pie charts, tables, image galleries, scatter plots, Venn diagrams, Euler diagrams)
– What do the data show?
– What is the authors’ interpretation of the data? What is YOUR interpretation of the data?
– Does the authors’ interpretation of the data seem valid? Does the interpretation go beyond what the data have actually shown?
– Have the authors considered alternative interpretations of their data? (Or is it a just-so story?)
– If so, have they tested any of these alternative interpretations?
– Can YOU think of an alternative interpretation of their data?
– Would YOU conclude the same thing they did from the data?
– Can YOU think of a more precise interpretation of their data? What have they actually shown?
– Have the authors strengthened their conclusion by obtaining the same conclusion using a different assay or different approach?
– What alternative assay would YOU use to strengthen their conclusion? (How many can you think of?)
– (style point) Does the text go in figure order (A, B, C, D, E) or does it jump around (A, C, B, E, D) and is harder to follow?
– Have they explained why they did the next experiment?
Repeat until the end of the Results section is reached.
– Could you follow the paper without consulting the supplemental figures, or do the supplements actually contain essential data? (This is a particular problem for journals that impose restrictive word/figure limits on authors)
Part 6: Discussion
– (style point) Do the main discussion points occur in the same order they do in the Results? (much easier to follow)
– Have the authors placed their results in the context of existing knowledge? Have they highlighted areas of agreement/disagreement?
– Have they acknowledged where alternative interpretations of their data are possible? Have they explained why they were not able to resolve these issues?
– Have they synthesised their results into a model of what they think is happening?
– Do YOU agree with the model? Does it accurately convey the data, or is it too speculative?
– Have they suggested how elements of their model could be tested?
– Do YOU agree with their conclusions?
Part 7: Materials and Methods
– Could YOU repeat their experiments based on the information provided? If not, what information is missing?
– Have they used appropriate terminology (“separated by centrifugation”, “amplified by PCR”, “bacteria were transformed using…”) or slang? (“spun down”, “PCRed”, “plasmid X was transformed into bacteria”)
– Have they indicated where they purchased or obtained non-standard reagents?
– Have they explained how data were quantified? (blinding, bias, determination of sample size, choice and justification of statistical tests, if used – almost NOBODY does this last point)
– Have they named the counterion for each pH buffer? (e.g. Tris-HCl pH 9, not Tris pH9) Are there any other minor errors?
Part 8: Overall
– Do YOU trust the paper? Would you be happy to put your name to it?
– If you do not concur with the authors’ conclusions or are unhappy with the data quality, what would YOU conclude about what has been learnt?
– How would YOU summarise the paper in 3-4 sentences?
– A key thing throughout is transparency. You should be able to look at the data, draw your own conclusions, and then check if your conclusions match those of the authors. If you draw different conclusions, or if you’re not able to draw conclusions, this is a sign that the paper is not optimally written/presented (again, this might be partly due to limits imposed by journal).
Remember that no paper is perfect, and if you want to find fault you always can do. Good critical reading is not about trashing other people’s work, it’s about trying to establish what the objective truth is, and either helping people improve or helping yourself improve by considering how things can/could be done.
Never, ever forget that the authors have invested a great deal of time, energy, and thought into their work – in most cases, lower-quality work is a result of inadequate training or low internal standards, not deliberate laziness. Your job as a critical reader is to identify areas of weakness, and suggest how they could be strengthened. Always treat authors with respect, even if you feel that they’ve wasted your time.
(This document originally written by Brooke Morriswood in May 2018 for students taking the Molecular Parasitology course at the University of Würzburg)