Custom fMRIPrep with Apptainer on Niagara

Custom fMRIPrep with Apptainer on Niagara

Overview

This guide outlines the steps for setting up a custom fMRIPrep container using Apptainer on Niagara, configuring Templateflow for offline use, and preparing a submission script for running fMRIPrep in the Niagara cluster environment.

1. Setting up the fMRIPrep Container

1.1 Verify Apptainer Installation

1. Ensure Apptainer is installed on Niagara. To verify, run:

which apptainer

A path like /usr/bin/apptainer should be returned.

2. If Apptainer is not enabled, enable it by running:

module load apptainer

1.2 Creating the Container

1. Create a definition file to build the container. In your desired directory, run:

cat <<EOF > fmriprep.def
From: nipreps/fmriprep:latest

%post
   apt-get update && apt-get install -y python3-pip
   pip3 install pysocks
EOF

2. Build the container with:

apptainer build fmriprep-latest.sif fmriprep-latest.def

3. Verify the container by running:

apptainer run fmriprep-latest.sif --version

This should print the version of fMRIPrep in the container.

2. Setting up Templateflow

fMRIPrep will try to download templates from the internet. To avoid this, we set up Templateflow locally.

2.1 Establishing the Templateflow Directory

1. Define the Templateflow home directory, for example:

export TEMPLATEFLOW_HOME=$HOME/templateflow

2. Add this to your .bashrc or .zshrc file:

echo "export TEMPLATEFLOW_HOME=$HOME/templateflow" >> ~/.bashrc

3. Create the directory if it does not exist:

mkdir -p $TEMPLATEFLOW_HOME

2.2 Using Python to Install Templateflow

1. Activate your Python environment. For instructions, see [these instructions](https://docs.scinet.utoronto.ca/index.php/Python#Using_Virtualenv_in_Regular_Python).

2. Install Templateflow:

pip install templateflow

3. Verify the TEMPLATEFLOW_HOME environment variable:

echo $TEMPLATEFLOW_HOME

4. Start a Python session to install templates:

python

Within Python, run:

import templateflow.api as tf
tf.templates()
tf.get(["MNI152NLin2009cAsym", "MNI152NLin6Asym", "OASIS30ANTs", "fsLR", "fsaverage"])

5. Exit Python with exit().

3. Establishing an SSH Tunnel to the Niagara Login Node

Establish this tunnel each time you log in to Niagara:

ssh -D 44223 nia-login02 -f -N

Replace nia-login02 with your current login node (check with echo $HOSTNAME).

4. Preparing a Script to Run fMRIPrep

Assuming you’ve set up everything above, have a BIDS dataset ready, and have a Freesurfer license file, you can use the following example script:

#!/bin/bash
#SBATCH --job-name=fmriprep
#SBATCH --output=specify/where/to/save/fmriprep.log
#SBATCH --error=specify/where/to/save/fmriprep.err
#SBATCH --time=12:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=40
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=your.email@whatever.ca

export APPTAINER_INSTANCE=/path/to/your/container/file
export BIDS_DIR=/path/to/your/bids/dataset
export FS_LICENSE=/path/to/your/freesurfer/license/file

if [ ! -d $TEMPLATEFLOW_HOME ]; then
    echo "Templateflow directory does not exist: $TEMPLATEFLOW_HOME"
    exit 1
fi

if [ ! -d $BIDS_DIR ]; then
    echo "BIDS directory does not exist: $BIDS_DIR"
    exit 1
fi

if [ ! -d $BIDS_DIR/derivatives/fmriprep ]; then
    mkdir -p $BIDS_DIR/derivatives/fmriprep
fi

if [ ! -f $FS_LICENSE ]; then
    echo "Freesurfer license file does not exist: $FS_LICENSE"
    exit 1
fi

export all_proxy=socks5://localhost:44223
export APPTAINERENV_TEMPLATEFLOW_HOME=/templateflow

apptainer run \
    --cleanenv \
    -B $BIDS_DIR:/data \
    -B $BIDS_DIR/derivatives/fmriprep:/data/derivatives/fmriprep \
    -B $FS_LICENSE:/freesurfer_license.txt \
    -B $TEMPLATEFLOW_HOME:$APPTAINERENV_TEMPLATEFLOW_HOME \
    $APPTAINER_INSTANCE \
    /data \
    /data/derivatives/fmriprep \
    participant \
    --random-seed 42 \
    --omp-nthreads 40 \
    --fs-license-file /freesurfer_license.txt

Make the script executable:

chmod +x your_script.sh

5. Running the Script

Submit the script with:

sbatch your_script.sh

Check the job status with:

sq

Good luck!