Overview

This guide outlines the steps for setting up a custom fMRIPrep container using Apptainer on Niagara, configuring Templateflow for offline use, and preparing a submission script for running fMRIPrep in the Niagara cluster environment.

Acknowledgments

Special thanks to Maurice Pasternak for providing this information and guidance on setting up the fMRIPrep container on Niagara.

Setting up the fMRIPrep Container

Verify Apptainer Installation

We will use Apptainer to run fMRIPrep. Apptainer should be installed by default on Niagara. To verify, run:

which apptainer

A path like /usr/bin/apptainer should be returned.

If Apptainer is not enabled, enable it by running:

module load apptainer

Creating the Container

We will create and use a definition file to create the container and add some additional Python packages to ensure compatibility in the event that fMRIPrep demands internet access.

This is a one-time setup.

Let's start by creating a definition file. Go to the directory where you want to create the container and run:

cat <<EOF > fmriprep.def
From: nipreps/fmriprep:latest

%post
   apt-get update && apt-get install -y python3-pip
   pip3 install pysocks
EOF

This will create a file called fmriprep.def. Use it to build the container with:

apptainer build fmriprep-latest.sif fmriprep-latest.def

This will create a file called fmriprep-latest.sif in the current directory. It is the fMRIPrep container of interest. You can verify the container by running:

apptainer run fmriprep-latest.sif --version

This should print the version of fMRIPrep in the container.

Setting up Templateflow

fMRIPrep will try to download templates from the internet. To avoid this, we will try to setup Templateflow locally and use environment variables to inform fMRIPrep where to find it.

This is a one-time setup.

Establishing the Templateflow Directory

Define the directory where Templateflow will be installed, for example:

export TEMPLATEFLOW_HOME=$HOME/templateflow

For convenience, you can add the above command to your .bashrc or .zshrc file which should be located in your home directory:

echo "export TEMPLATEFLOW_HOME=$HOME/templateflow" >> ~/.bashrc

If this directory we have indicated does not exist, create it:

mkdir -p $TEMPLATEFLOW_HOME

Using Python to Install Templateflow

You can either activate a Python environment you have used previously or create a new one. For instructions, see these steps.

Now, let's install Templateflow.

pip install templateflow

Make sure the TEMPLATEFLOW_HOME environment variable is set correctly:

echo $TEMPLATEFLOW_HOME

This should point to the directory you created earlier. If this prints nothing, refer back to the section on establishing the Templateflow directory.

Assuming the above checks out, let's start up a Python session and install the desired templates.

python

Within Python, run:

import templateflow.api as tf
tf.templates()
tf.get(["MNI152NLin2009cAsym", "MNI152NLin6Asym", "OASIS30ANTs", "fsLR", "fsaverage"])

This will take a while to run. After it is complete, try to re-run the get command again. It should return immediately with a list of Path objects, with no download taking place since the templates have already been downloaded.

Let's exit the Python session by typing exit().

Establishing an SSH Tunnel to the Niagara Login Node

This must be done every time you log in to Niagara if you intend to use the container.

It is possible that fMRIPrep will at some point request internet access. To avoid this, we will establish an SSH tunnel to a Niagara login node:

ssh -D 44223 nia-login02 -f -N

Where nia-login02 is an example login node. Preferably, replace with the login node that you are currently using. You can check which login node you are using by running echo $HOSTNAME and looking at the nia-login## part.

Preparing a Script to Run fMRIPrep

Assuming you’ve set up everything above, have a BIDS dataset ready, and have a Freesurfer license file, you can use the following example script:

Followed the instructions above
Have a BIDS dataset ready on scratch or burst buffer
Have a freesurfer license file placed in a known location

Here would be an example script that you can use to run fMRIPrep:

#!/bin/bash
#SBATCH --job-name=fmriprep
#SBATCH --output=specify/where/to/save/fmriprep.log
#SBATCH --error=specify/where/to/save/fmriprep.err
#SBATCH --time=12:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=40
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=your.email@whatever.ca

export APPTAINER_INSTANCE=/path/to/your/container/file
export BIDS_DIR=/path/to/your/bids/dataset
export FS_LICENSE=/path/to/your/freesurfer/license/file

if [ ! -d $TEMPLATEFLOW_HOME ]; then
    echo "Templateflow directory does not exist: $TEMPLATEFLOW_HOME"
    exit 1
fi

if [ ! -d $BIDS_DIR ]; then
    echo "BIDS directory does not exist: $BIDS_DIR"
    exit 1
fi

if [ ! -d $BIDS_DIR/derivatives/fmriprep ]; then
    mkdir -p $BIDS_DIR/derivatives/fmriprep
fi

if [ ! -f $FS_LICENSE ]; then
    echo "Freesurfer license file does not exist: $FS_LICENSE"
    exit 1
fi

export all_proxy=socks5://localhost:44223
export APPTAINERENV_TEMPLATEFLOW_HOME=/templateflow

apptainer run \
    --cleanenv \
    -B $BIDS_DIR:/data \
    -B $BIDS_DIR/derivatives/fmriprep:/data/derivatives/fmriprep \
    -B $FS_LICENSE:/freesurfer_license.txt \
    -B $TEMPLATEFLOW_HOME:$APPTAINERENV_TEMPLATEFLOW_HOME \
    $APPTAINER_INSTANCE \
    /data \
    /data/derivatives/fmriprep \
    participant \
    --random-seed 42 \
    --omp-nthreads 40 \
    --fs-license-file /freesurfer_license.txt

Save the script .sh file to a known location and remember to ensure it is executable:

chmod +x your_script.sh

Running the Script

Submit the script to the queue with:

sbatch your_script.sh

You can check the status of the job with:

sq

Good luck!

Custom fMRIPrep with Apptainer on Niagara

Contents