Ten Top Tips for Reviewing Statistics: A Guide for Ecologists

post provided by Dr Mark Brewer.

Mark is a statistician with Biomathematics & Statistics Scotland, based in Aberdeen. His main statistical research interests are Species Distribution Modelling, Compositional Data Analysis, Bayesian Mixture Modelling and Bayesian Ordinal Regression. Mark was one of the presenters at the UK half of the Methods in Ecology and Evolution 5th Anniversary Symposium in April. You can watch his talk, ‘Model Selection and the Cult of AIC’ here.

The level of statistical analysis in ecology journals is far higher than in most other disciplines. Ecological journals lead the way in the development of statistical methodology, necessitated by challenging practical problems involving complex data sets. As a statistician, publishing also in hydrology, soil science, social science and forensic science journals, I’ve found papers in those areas are much more likely to only use well-established methods than papers in ecology.

Here’s the big question though: why then do I have the most difficulty with ecological journals when it comes to statistical analyses? Let’s be clear here: when I say “difficulty”, I mean I receive reviews which are just plain wrong. Most statisticians I’ve spoken to who work in ecology have anecdotes from reviews which demonstrate a lack of understanding by the non-statistician reviewer (including the all-too-frequent “perhaps you should consult a statistician”). So, why the apparent disconnect?

The difference seems to be in how non-statisticians in different disciplines treat the statistics in a paper. In many subject areas, reviewers are almost deferential to the statistical analysis; in ecology, reviewers can be forthright in their condemnation, often without justification. Reviewers have every right to question the statistical analysis in a paper, but the authors have the exact same right to expect a high quality review from a genuine expert in the field. Has ecology become blasé about statistics?

To address this, I would like to offer the following suggestions to ecologists reviewing the statistical content of papers in ecology:

1. Be Honest about What You Know and What You Don’t

I know relatively little about statistical community analysis; the experts I know are ecologists, not statisticians. So I’d be cautious about claiming too much expertise in the area, and I’d say so in the review. In one case, we had a reviewer (who we assume was an ecologist rather than a statistician) tell us we should have used a mixed model; given that there were no factors in the data set, we had no idea how this was even going to be possible! When I’m reviewing a paper in an ecological journal, at the beginning I state clearly that I am a statistician, and that I will be focussing on the stats – I’ve worked with ecologists for a number of years now, but I’m no ecologist. Making a clear early statement about your areas of expertise helps the editors with their decisions and the authors with their responses to your recommendations.

2. Old Doesn’t Mean Bad

I’ve lost count of the number of times I’ve been told the method I’ve used is “out-of-date” or “old-fashioned”. It doesn’t matter that, say, logistic regression was formalised over 50 years ago – it is still a perfectly valid method. Age should never be used to criticise a method per se.

Old doesn’t mean bad: The face of modern, Bayesian statistical methodology.

3. New Doesn’t Mean Good

The counterpoint to old doesn’t mean bad. That fancy new method might make fewer assumptions, but be harder to fit, so be wary of insisting it be used. It might be more data-hungry, meaning that it’s not feasible for the data set in the paper; this can manifest itself in things like massively inflated standard errors caused by problems with numerical optimisation. These are things to look for, as well as the more obvious errors; for example I’ve seen people use complex non-parametric smoothing on five data points…

4. Understand Assumptions

Any statistical method involves making assumptions. It is important to understand what those assumptions are before you judge whether the method is being used appropriately. For example, is it valid to assume a straight-line relationship between response and covariate in a regression? Or would a simple parametric relationship suffice? Only if the answer to both questions is “no” would I expect to see an advanced model using a smoother (for example, a generalised additive model) as part of the final results.

Understand assumptions: When ordering “pizza and chips”, not understanding assumptions can lead to surprising conclusions.

A good general rule is to try to use the simplest possible method whose assumptions are valid in any analysis; why make life difficult for the sake of it? Suggesting unnecessarily complicated methodology makes things harder for the authors, the editors, for you as the reviewer (when the revised paper comes back), and finally, (if published) the readers. Doing that isn’t impressive, it’s foolish.

5. Match the Method to the Assumptions

The trick is to look at the method used in the paper, to understand what assumptions are made by the method and then to use your expert knowledge to judge whether those assumptions are reasonable. If the assumptions are not stated clearly, you’re well within your rights to ask that they are in a revised version. This is a process which should involve both statisticians and ecologists; I know what assumptions a method makes, but I need an ecologist’s input on how justifiable those assumptions are in practice for any given example.

6. There is Rarely a Single Correct Method

There might be doubt about what assumptions are reasonable in any analysis. Just because the assumptions you’re prepared to make differ from the authors’, it doesn’t always follow that either of you are wrong. Are the authors’ assumptions completely unjustifiable? You should only insist on a different statistical method if the answer is “yes”.

7. p-values are not the Work of the Devil

They really aren’t. That’s my claim, and I’m a Bayesian. A paper should not be rejected just because it uses p-values and, as confidence intervals are just p-values rearranged, recommending them isn’t going to change anything. Of course, if a paper contains p-values or AIC statistics aplenty but not a single effect size, then that’s a different story. But p-values are at least sensible if a paper is setting out to test or examine a small number of clearly defined hypotheses.

8. Beware of Uninformed Pronunciamentos*

Occasionally we see articles or editorials proclaiming a statistical approach is worthless. A recent editorial in a psychology journal banned most statistical inference (leaving the door ajar only to possibly allow some Bayesian analysis to try to sneak through). Elsewhere, in ecology journals some authors have gone to great lengths to claim that accounting for things like spatial autocorrelation or detection probability is unimportant. In reality, things are almost never that simple, so beware of believing in papers or books which denounce for all time some statistical method; the rebuttal is usually not far behind.

*Thanks to Stuart Hurlbert for this marvellous phrase

9. Recommend a Statistician Look at the Paper

There’s no shame in this! Some journals have statistical “panels” who get called upon to look at papers only if a query has been raised by one of the reviewers, or if an editor feels there are issues with the paper which require input from a statistician (with medical journals it’s not uncommon for statistical reviewers to be paid). The problem in ecology may be that there are so many papers with substantial statistical content that, for many journals, a statistical panel could in theory see almost all papers submitted and the role would be unmanageable. In practice, the panel might perhaps only be drawn upon in the most contentious cases or where the reviewers and authors are having an especially “lively” disagreement. Bringing in a statistician early on can make the process much smoother for everyone involved.

10. Read up on the Statistical Methods

Are you entirely confident you fully understand the stats in the paper? It cannot hurt to check your own understanding during the review process. There may be new research into the methodology you weren’t aware of – so use the opportunity afforded you by reviewing to refresh your own knowledge, or even to learn something entirely new. Then you can make a proper, valid assessment of the statistical content of the paper.

Hopefully these ten tips will help bring the level of statistical reviewing in ecology up to the high standards of statistical analysis which can be found in ecological journals.

Thanks to Alison Johnston for helpful comments on an early draft of this post and, in particular, for suggesting tip #10.

Watch Mark’s presentation on ‘Model Selection and the Cult of AIC’ at the Methods in Ecology and Evolution 5^th Anniversary Symposium here.