Computing with Flywheel

May 13, 2022

Travis Richardson

By Travis Richardson

I am often asked to explain how Flywheel supports a broad range computational workflows, including:

Working with existing pipelines
Exploratory development and analysis
Automating routine processing

Flywheel offers an open and extensible approach that provides you the flexibility to work in the manner that makes sense for your lab or project.

Working with existing processing pipelines

The simplest approach for working with existing pipelines involves downloading the required data from Flywheel and processing it as usual. Flywheel provides several download options including the web-based UI and command line tools. For more control over selecting and formatting data, Flywheel provides easy-to-use programming interfaces for use with leading scientific languages including Python, MATLAB, and R. These may be used to access, format, and download any data or metadata in the Flywheel database.

Exploratory Development and Analysis

For developing new algorithms or pipelines, Flywheel’s Python and MATLAB SDKs provide a powerful alternative to downloading to disk. Using the SDKs, a Python or MATLAB user may work with data in Flywheel directly from their preferred scripting language. Full search is available along with simple commands for reading and writing data and metadata.

Routine Processing with Plug-In Applications (Gears)

Gears are plug-in applications that automate routine tasks, including metadata extraction, classification, quality assurance, format conversion, and full analytic pipelines. Here’s how gears work:

Leveraging Standard OCI-Compliant Containers

From a technical perspective, Gears are applications running in standard OCI-compliant (Docker, Singularity, etc.) containers that are managed by Flywheel. A container typically contains application code and all of its dependencies to create a portable, reproducible unit of processing. Containers can be easily made into Gears with the addition of metadata that explains to Flywheel how to use the containerized applications. This metadata is expressed via a simple JSON file that includes descriptive metadata, such as links to source code, authors, etc. It also includes instructions for passing in data, configuration options, and how to execute commands in the container.

Automating and Scaling Gear Execution

Gears may be run in a variety of ways. They may be executed on demand for a given data set. They may also be run in batch mode for a selected collection of data sets. In these cases, the user is prompted for inputs prior to execution. Gears may also be run automatically by rules configured for the project. For example, when a DICOM series is uploaded, it can be classified and converted to NIfTI, if it is imaging data. Gear rules may be used to automate routine pre-processing as well as trigger complex pipelines. Gears may be scheduled by tasks outside of Flywheel using the command line tool (CLI) or programming interfaces. Finally, when deployed in cloud or private cloud infrastructures, Flywheel can dynamically scale resources to maximize parallel processing to save you time.

Process Any Level of Data in Your Project

Gears may be designed to process data at different levels of the Flywheel project hierarchy. Gears may process individual sessions (exams/DICOM studies). For longitudinal studies, Gears may be used to process at the subject (participant/patient) level with the ability to process data from multiple sessions. Finally, project-level Gears may be used to perform group/cohort analyses across all subjects.

Automated Provenance

A key advantage of using Gears to manage routine processing is the documentation that results. Everytime a Gear is run, Flywheel records a great deal of derivative information that supports consistency and reproducibility of your project. These “Analysis” documents record Gear version, who ran it, when it ran, success/fail status, inputs, configuration options used, and outputs produced. Further, they may be annotated with notes or structured JSON metadata to meet your project needs. This provenance makes it easy to ensure that all necessary processing steps were performed and performed consistently.

Flywheel Gear Exchange

To speed project deployment, Flywheel provides a library of commonly used algorithms as Gears via the Flywheel Gear Exchange. The Gear Exchange currently contains roughly 70 Gears contributed by Flywheel or Flywheel users. Examples include DICOM-to-NIfTI conversion, Freesurfer Recon-All, the Human Connectome Pipelines, and commonly used BIDS applications, such as MRIQC and FMRIPrep. The Gear Exchange provides a powerful way to share reproducible units of code that may be used as building blocks for new projects.

User-Developed Custom Gears

Users may easily create their own Gears as well. Gear developers simply get their code running in an OCI-compatible container and provide the gear metadata. Applications may be developed in any language. Flywheel’s APIs and SDKs may be used in a Gear if needed, otherwise, the containerized application need not be Flywheel aware.

Flywheel streamlines the process of creating the Gear metadata via the CLI Gear Builder tool which prompts the user through the required information and generates most of the metadata automatically. The resulting Gears may be shared with other Flywheel sites via the Flywheel Gear Exchange, or may be kept private by uploading them only to the user’s site. Flywheel does not make any claim on any of the intellectual property in customer Gears.

Conclusion

Flywheel makes it easy to work the way you want. Our open CLI, APIs, and SDKs make it easy to download data and use existing processes. Our Gears framework allows you to automate routine processing consistently with extensive documentation to support quality and reproducibility.