The role of science journals is to publish papers about scientific research. We need to maintain some quality in what is published, so we use peer review, and ask experts in the subject of a paper to read it and check that it is correct, the arguments make sense etc.
One of the types of paper we publish is Applications, most of which describe software that will help ecologists and evolutionary biologists to do their research. Our focus is on the paper itself, but we also want to be confident that the software is well written, e.g. that it has no obvious bugs, and that it is written so that future versions will not break.
Of course, it takes a lot of time to thoroughly review software, and that is not the primary job of the journal’s peer review process. But we appreciate that this needs to be done, and indeed many of our reviewers and editors put a lot of time into doing just this, something we really appreciate. But can we do this better?
Fortunately, we were approached by the rOpenSci organisation, who wanted to collaborate with us to do this (a huge thanks to Scott Chamberlain for this initial approach and all of his hard work in putting this collaboration together). They are a group of coders, mainly in ecology, who have written a large number of open source R packages for a variety of tasks (e.g. importing data, visualisation). They also want to maintain good quality code, so they have implemented a variety of methods to do this.
One of these is code review. This is another form of peer review, but focused on the code, not the paper. This means the reviewer can concentrate on checking that the code works, that it is well written and documented (so other people can read the code and adapt it), and that it has the right sets of tests, so that if something changes, it is straightforward to check that it still works.
After discussion, we have now worked out a protocol to combine peer review of the paper and the code for R Applications manuscripts submitted to Methods in Ecology and Evolution. For manuscripts that are within the scope of both rOpenSci and the journal, what will happen is:
- Code will be submitted to the rOpenSci Onboarding platform on GitHub.
- In the submission form, the should authors note that they intend to submit a manuscript about the software to Methods in Ecology and Evolution.
- Once the code has been reviewed by rOpenSci, the authors are encouraged to submit a manuscript about it to the journal. Authors can submit their manuscript once the reviewer comments on the code have been received or they can choose to wait until the package has been approved.
- Associate Editors will be made aware that the code has been accepted or is under review with rOpenSci, so that they can be confident the code works as advertised in the manuscript.
- Review process proceeds as normal in Methods in Ecology and Evolution, but this should be quicker than usual as we won’t need to review the code associated with the paper.
- If the paper is accepted for publication, a link to the code in rOpenSci will be provided in the ‘Data Accessibility’ section of the manuscript along with a note on the first page of the manuscript to confirm that the code has been accepted by rOpenSci.
We’re looking forward to seeing how this works. Our hope is that it will benefit the journal (because we will publish more papers that describe high quality code), the writers of the Applications (who will get advice on how to improve their code), and rOpenSci (who will enlarge their community). And this will all benefit the end user, who will be able to use better code, and be confident that it has been double-checked and works as advertised, and works well.
Methods Blog 
6 thoughts on “Software Review Collaboration with rOpenSci”