Lerma-Usabiaga led a group from Stanford University and the University of California, San Francisco (UCSF) in developing a framework for translating magnetic resonance imaging (MRI) scientific findings into clinical practice. Their system is based on the Flywheel neuroinformatics platform, including both the data and computational management tools. The methods framework explores replication, which is essential for valid science, and generalization, which is essential for clinical applications.
The authors gathered nine data sets into the Flywheel neuroinformatics platform, grouping them into three categories. Variations in data characteristics begin during acquisition, the authors note, due to calibration differences in MRI instruments between competing MRI vendors and other factors.
Neuroinformatics platform reproducibility
Lerma-Usabiaga, et al., (2019) use Flywheel to support their goal of computational reproducibility. They note that computational reproducibility is supported by using open source containerized methods whose inputs, outputs, parameters and versioning (provenance) are stored in the Flywheel neuroinformatics platform. Other scientists can reproduce the analysis by accessing the Flywheel system, if they are authorized by the Institutional Review Board (IRB).
The containers execute the largest and most complex jobs, while Flywheel’s software development kit (SDK) facilitates data preparation, further statistical analysis, and visualization (2019, p. 4). Script reproducibility is supported via storage and versioning in a GitHub repository (https://github.com/garikoitz/paper-reproducibility) while input data and the executed version is stored in the Flywheel neuroinformatics platform.
Neuroinformatics platform for scientific reproducibility
The specific application the authors explore is a potential biomarker in the white matter that can be used to assess individual subjects, following the “Precision Medicine” approach emerging in neuroimaging. The authors point to a need for quantitative and objective method frameworks addressing biomarker measurement validation which provides precision, as well as replication, and reproducibility.
The paper addresses increasing variations in the clinical setting – neuroimaging instruments, populations, and measurement protocols – as well as the problem of algorithmic complexity. Lerma-Usabiaga, et al., (2019) point out that by using containers and storing the analysis history the Flywheel neuroinformatics platform system implemented “closely align” (2019, p. 8) with the Poldrack et al., (2017) description of scientific reproducibility research tools
“…entire analysis workflow…completely automated in a workflow engine and packaged in a software container or virtual machine to ensure computational reproducibility.” (p. 124, 2017)
Finally, Lerma-Usabiaga, et al., 2019 note that the Flywheel platform is extensibile, simplifying the analysis of new datasets using identical computational methods and measuring how compliance ranges fluctuate. A process of continuous data aggregation should allow continuing improvement of the methods and better definition of the compliance range, helping translate scientific research from the lab to the clinic.
Lerma-Usabiaga, G., et al., 2019. Replication and generalization in applied neuroimaging. NeuroImage, a Journal of Brain Function. Volume 202, Article 116048, November 15, 2019.
Marcus, D., et al., 2011. Informatics and data mining tools and strategies for the human connectome project. Front. Neuroinf., 5 (2011), p. 4.
Poldrack, R. A., et al., 2017. Scanning the horizon: towards transparent and reproducible neuroimaging research. Nat. Rev. Neurosci. 18, 115-126.