Leveraging Flywheel for Deep Learning Model Prediction

Since 2012, the Medical Image Computing and Computer Assisted Intervention Society (MICCAI) has put on the Brain Tumor Segmentation (BraTS) challenge with the Center for Biomedical Image Computing and Analytics (CBICA) at the Perelman School of Medicine at the University of Pennsylvania. The past eight competitions have seen rapid improvements in the automated segmentation of gliomas. This automation promises to address the most labor-intensive process required to accurately assess both the progression and effective treatment of brain tumors.

In this article, we demonstrate the power and potential of coupling the results of this competition with a FAIR (Findable, Accessible, Interoperable, Reusable) framework.  With constructing a well-labeled dataset constituting the most labor-intensive component of processing raw data, it is essential to automate this process as much as possible. We utilize Flywheel as our FAIR framework to demonstrate this process.

Flywheel (flywheel.io) is a FAIR framework that leverages the proprietary core infrastructure with open-source extensions (gears) to collect, curate, compute on, and collaborate on clinical research data. The core infrastructure of a Flywheel instance manages the collection, curation, and collaboration aspects, enabling multi-modal data to be quickly searched across an enterprise-scale collection. Each “gear” of the Flywheel ecosystem is a container-encapsulated open-source algorithm with a standardized interface. This interface enables consistent stand-alone execution or coupling with the Flywheel core infrastructure—complete with provenance of raw data, derived results, and usage records.

For the purposes of this illustration, we wrap into a gear the second-place winner of the MICCAI 2017 BraTS Challenge. This team’s entry is one of the few that has both a docker hub image and a well-documented github repository available. Their algorithm is built around both TensorFlow and NiftyNet frameworks for training and testing their Deep Learning model. As illustrated in our github repository, this “wrapping” constitutes providing the data configuration expected by their algorithm and launching their algorithm for model prediction (*).

As shown in the figure above, Flywheel provides a user-friendly interface to navigate to the MRI images expected for execution. With the required co-registered and skull-stripped MRI modalities (T1-weighted, T1-weighted with contrast, T2-weighted, and Fluid Attenuation Inversion Recovery), segmentation into distinct tissues (normal, edema, contrast enhancing, and necrosis) takes twelve minutes on our team’s Flywheel instance (see figure below). This task can take a person over an hour to segment the same tumor. When performed on a Graphical Processing Unit (GPU), this task takes less than three minutes to complete.

Segmentation into normal, edema, contrast enhancing, and necrosis tissues with the Flywheel-wrapped second place winner of the 2017 BraTS Challenge.

Although this example predictively segments the tumor of a single patient, modifications to this gear can allow tumor segmentation of multiple patients for multiple imaging sessions over the course of their care. Furthermore, with scalable cloud architecture, these tasks can be deployed in parallel, significantly reducing the overall time required to iterate inference over an entire image repository. Enacting this as a pre-curation strategy could significantly reduce the time necessary for manual labeling of clinical imaging data. 

Therein lies the vast potential benefit from using a strong FAIR framework in an AI-mediated workflow. Being able to pre-curate new data, optimize human input, and retrain on well-labeled data over accelerated time-scales. These model design, train, and test cycles are greatly facilitated by a FAIR framework, which is able to curate the data, results, and their provenance in a searchable interface.

As with this brain tumor challenge example, there are many other similar challenge events that make their algorithms and pretrained models publicly available for the research community.  One nexus of these is the Grand Challenges in Biomedical Image Analysis, hosting over 21,000 submissions in 179 challenges (56 public, 123 hidden).  Flywheel’s capacity to quickly package these algorithms to be interoperable with its framework makes it a powerful foundation for a data-driven research enterprise.

Two more useful deep learning and GPU-enabled algorithms have recently been incorporated into Flywheel gears. First, quickNAT uses default or user-supplied pre-trained deep learning models to segment neuroanatomy within thirty seconds when deployed on sufficient GPU hardware. We have wrapped a Pytorch implementation of quickNAT in a Flywheel gear. Prediction of brain regions on CPU hardware requires two hours.  Although much longer than thirty seconds needed on a GPU, it is still a fraction of the nearly twelve hours needed for FreeSurfer’s recon-all. Next, we have Nobrainer, a deep learning framework for 3D image processing. The derived Flywheel gear uses a default (or user-supplied) pre-trained model to create a whole brain mask within two minutes on a CPU. Utilizing a GPU brings this time down under thirty seconds.

The previous paragraph elicits two questions. First, with GPU model prediction times significantly faster than CPUs, when will GPU-enabled Flywheel instances be available? The next being, how can Flywheel be effectively leveraged in training deep learning models? Flywheel is actively developing GPU-deployable gears and the architecture to deliver them.  We briefly explore the second question next, leaving a more thorough investigation for another article.

Training on an extensive and diverse dataset is needed for Deep Learning models to generalize effectively and accurately across unseen data. With uncommon conditions, such as gliomas, finding enough high-quality data at a single institution can be daunting. Furthermore, sharing these data across institutional boundaries incurs the risk of exposing protected health information (PHI). With Federated Training, Deep Learning models (and their updates) are communicated across institutional boundaries to acquire the abstracted insight of distributed annotation. This eliminates the risk and requirement of transferring large data repositories while still allowing model access to a diverse dataset. With Federated Search across institutional instances of Flywheel firmly on the roadmap, this type of Federated Training of Deep Learning models will be possible within the Flywheel ecosystem.

(*) The authors of this repository and the University College London do not explicitly promote or endorse the use of Flywheel as a FAIR framework. 

Flywheel-Connect 3D Slicer Extension

Modern scientific workflows require the utilization of a diverse (and sometimes disparate) set of tools to turn data into insight. With each of these tools designed to excel at a specific set of tasks, shepherding results between tools can require significant effort and expense. Fortunately, Flywheel is an integrated framework facilitating ease-of-use of these tools and the transfer of data between them. The Flywheel-connect tool described here is an 3d Slicer extension which further enhances the Flywheel platform's integrated functions. 

Extending the Flywheel-Connect 3D Slicer Extension

In this article we demonstrate the capacity of Flywheel to extend the utility of 3D Slicer. 3D Slicer is an open source software platform for medical image informatics, image processing, and three-dimensional visualization. While 3D Slicer excels at desktop visualization and local image analysis, it lacks the capacity to orchestrate cloud-centric computing and institutional data resources -- the specific domain in which the Flywheel 3D Slicer extension shines. 

We have leveraged 3D Slicer’s Extension architecture and Flywheel’s Python SDK to provide 3D Slicer access to NifTI images stored within a remote Flywheel instance running on Google Cloud Platform. Called “flywheel-connect”, this 3D Slicer extension is a proof-of-concept utility for downloading and displaying multiple NifTI images from selected acquisitions within a specific Flywheel instance. The “flywheel-connect” extension makes it possible for a 3D Slicer extension user to directly access and load data stored within any Flywheel instance they have access to.


We demonstrate flywheel-connect in the video below using data from the 2017 MICCAI Multimodal Brain Tumor Segmentation Challenge. Firstly, a user-generated api-key from a Flywheel instance is input.  Then, after the “Connect Flywheel” button is pressed, combo-boxes are populated with all of the groups, projects, sessions, and acquisitions that the user has permissions to access. To load an image into the 3D Slicer extension, press “Retrieve Acquisition”. By default, images are cached in a user-specific directory for later use. These directories and images can be deleted directly off of the filesystem or flushed with the “Cache Images” checkbox.

When the data is in 3D Slicer, Slicer-specific operations can be performed on these data. As portrayed in the video, the discrete-valued volume representing tumor segmentation across imaging modalities (FLAIR, T1W, T1CE, T2W) is converted into a label map.  In turn, this label map is converted to a three-dimensional representation with Slicer-provided tools.

Although flywheel-connect demonstrates a usable solution, additional features would greatly increase overall utility for Slicer-centric imaging workflows. First, the capacity to view the results of specific Flywheel analyses. This would, for example, allow the visual inspection of image registration to an atlas or template done in Flywheel. Secondly, saving an entire 3D Slicer workspace in a Flywheel instance.  Along with instantiating this workspace from a Flywheel instance, this feature would provide functionality entirely lacking in 3D Slicer: Management of multiple workspaces relating to the same project.

To try out flywheel-connect for yourself, see the github page for this project. Suggestions and proposed improvements are welcome. https://github.com/flywheel-apps/flywheel-connect