Post provided by Dr Eleni Matechou
Eleni is a Lecturer in Statistics and a member of the Statistical Ecology @ Kent (SE@K) group at the University of Kent. She develops statistical models motivated by ecological applications to study populations of birds, insects and, more recently, humans.
On September 15th 2016, a half-day meeting on Model Averaging – organised by the Environmental Statistics section and the East Kent local group of the Royal Statistical Society (RSS) – took place at the University of Kent in Canterbury .
There were three invited speakers: Professor Richard Chandler, from University College London, Professor Jonty Rougier, from the University of Bristol and Dr Kate Searle, from the Centre for Ecology and Hydrology, who presented via Skype.
All three talks included interesting motivating data, clever modelling and great insight.
Taming the Pachyderm
Professor Richard Chandler presented joint work with Marianna Demetriou on “The interpretation of climate model ensembles”. Projecting future global temperatures is clearly a timely topic and Richard’s talk highlighted the challenges of doing this reliably. And they’re certainly not minor challenges, in his own words, this is a problem he has spent 10 years thinking about!
As Richard explained, there are different simulators that produce different projections, so researchers are advised to use multi-model ensembles (MME) to combine information from several simulators. However, there’s an elephant in the room!
Thankfully, Richard proposed ways in which the problems can be addressed and the pachyderm can be tamed! These formally model DEATH (!), which is the discrepancy of simulators from the truth. Proposed methods, ranging from the “poor man’s” to a fully Bayesian approach vary in their rigorousness and complexity but they provide similar results, albeit with some differences in the tails of the distributions.
Before closing, Richard presented some of the work from Marianna’s PhD thesis, which accounts for the fact that simulators come in families or clusters. The results are very similar to the existing ones, but as Richard put it, you wouldn’t know this without doing it!
And the Oscar goes to….
Professor Jonty Rougier presented on “Ensemble averaging and mean squared error” which is the title of his recent paper in the Journal of Climate.
The work was motivated by Figure 9.7 of the 5th assessment report of the IPCC (Intergovernmental Panel on Climate Change). The figure compares different simulators, represented by the columns, in their projections for several variables, represented by the rows. The colour of the rectangles shows whether the particular simulator performed better (blue colours) or worse (red colours) than the other simulators. As Jonty explained, this figure is the Oscars of simulators! However, no simulator can outperform all simulators for all variables as each one of them is “trained” on specific variables.
So, can you spot the issue!?
Well, I think most of us didn’t either until Jonty pointed it out to us! The ensemble mean, represented by the first column, not only does better than more than half of the simulators, i.e. the median, as the rectangles are mostly blue, it often also outperforms all of the simulators, which is shown by the deep blue rectangles.
According to Jonty this could be explained by a model for the simulator biases.
This states that if the bias of each simulator, µi, is smaller in absolute value than the standard deviation of the distribution of the differences between the simulator projections and the observations, σ (otherwise called the pixel errors), then the rank of the multi-model ensemble (MME) mean squared error, M bar, will tend to 0. That is the MME will outperform all simulators, as the number of pixels (variables) tends to infinity. As Jonty puts it in his paper “If individual simulators are tuned more on their overall bias than their large pixel errors, then we might expect something similar to this condition to hold.”
Jonty’s simulation of the “bad” situation, in which the above condition between bias and pixel errors does not hold, for small n (that is when the asymptotics do not actually hold either), suggests that “it is highly probable that the ensemble mean will perform far better than the median simulator”.
What’s Harder than Rocket Science? Ecology!
Dr Kate Searle’s talk, titled “Analysis of ecological data: Ecology isn’t rocket science, it’s harder!”, presented her journey of modelling messy, spiky and complex data on counts of midges. Because of their link to several new and devastating vector-borne diseases worldwide we need to know where, when and how many there are…
However, as Kate found out, this is not easy! Together with her collaborators – Adam Butler from BIOSS and Beth Purse from CEH – they have tried everything from GAMs to truncated normal mixture models with varying levels of success.
Currently, they’re thinking of reversible jump (RJ)MCMC for dealing with the unknown number of mixture components where each component represents a peak in the abundance of midges.
Finally, Kate listed the issues that ecologists face when it comes to model averaging. As she explained, she would not think of RJMCMC as a model-averaging tool, and she believes that most ecologists link model averaging with the use of covariates. The (mis)use of AIC and AIC weights was on the top of the list of issues as was the belief that one has to average between models.
She closed by suggesting that more dialogue is needed between ecologists and statisticians – well I think we can all agree with that!
As expected, this was a very interesting and informative meeting which highlighted the challenges but also opportunities in terms of research potential that exist in the area of model-averaging for both statisticians and practitioners.
Methods Blog 