What we have learned from the Reproducibility Project: Cancer Biology
Atualizado: 1 de Nov de 2018
Last week, Science Magazine reported that the Reproducibility Project: Cancer Biology (RP:CB) will go through its third large downsizing. The 50 articles initially selected for replication were reduced to 37 in 2015, 29 in 2017 and are now down to 18. Since the conception of the Brazilian Reproducibility Initiative (Brazilian Reproducibility Initiative), we have been inspired by the RP:CB and followed its development closely. With this in mind, we have designed our own project in a way that tries to avoid some of the obstacles faced by this and other large reproducibility projects (we will write more about them in a series of posts a few weeks from now).
Most of the difficulties faced by the RP:CB come from their choice to evaluate the reproducibility of high-impact articles, which often contain a large number of very complex experiments. That way, the inability to replicate, or even execute, some of the experiments due to technical complexities can hinder interpretations on the reproducibility of their findings. To try to avoid this problem, the Brazilian Reproducibility Initiative has chosen to focus on individual experiments from Brazilian articles, as each experiment will have a quantitative, clearly defined result. Moreover, we chose to include only commonly used techniques, which many labs across the country can perform, in order to replicate each experiment in at least 3 different labs. After an initial systematic review, our candidate techniques at the moment include MTT assays, TBARS assays, Western blot, qPCR, ELISA, flow cytometry, microscopy/HE staining, immunohistochemistry, elevated plus maze and the open field task.
The RP:CB took care to perform each replication with a protocol as close as possible to that used in the original article. This was not an easy feat, as the descriptions available in the articles were incomplete and many of the original authors did not respond to requests for methodological details. After discussing these issues with Tim Errington, the RP:CB's coordinator, we decided to take a more naturalistic approach. The methodological details of our replication will be based on what is reported in the original articles, which will inevitably be lacking in detail. After extracting what is available, we will send the incomplete protocols to the three teams involved in each experiment, which will independently fill in the gaps according to their own references and experiences. In this way, we will have a naturalistic measure of reproducibility based on what is written in an article, with no need for additional information from the part of authors.
Another limiting factor in RP:CB was obtaining specific materials that were developed by the original researchers for experiments. To avoid this pitfall, we will include in our selection only experiments that can be performed with commercially available materials. In our first systematic review, for example, we found a high prevalence of experiments evaluating the effect of natural extracts, obtained through processing of plants from diverse origins. Because those experiments would add an extra layer of complexity and an additional source of variability, we opted for their exclusion from our replication sample.
At this moment, the Brazilian Reproducibility Initiative is starting the systematic reviews for each of the candidate techniques. The aim of these reviews is to evaluate the methodological description of the individual steps of candidate experiments, which are being drawn from a random sample of Brazilian articles. These experiments will be later selected for replication, depending on the availability of research teams to replicate experiments on each technique (by the way, our registration call for collaborators will start soon!). The data obtained at this stage will provide important information about how each step is typically performed and about the prevalence of reporting of their methodological details in Brazilian articles.
You can participate in the Brazilian Reproducibility Initiative if you work in a research lab located in Brazil and possessing the required infrastructure to perform any of the candidate techniques. You can also contribute by participating in public consultations on the forms used to describe the experiments, which will also be available soon. Please subscribe to our newsletter to receive updates when the next steps are starting and to keep in touch with our latest blog posts!
By Clarissa Carneiro