Custom fMRIPrep with Apptainer on Niagara

Custom fMRIPrep with Apptainer on Niagara

Overview

This guide outlines the steps for setting up a custom fMRIPrep container using Apptainer on Niagara, configuring Templateflow for offline use, and preparing a submission script for running fMRIPrep in the Niagara cluster environment.

1. Setting up the fMRIPrep Container

1.1 Verify Apptainer Installation

1. Ensure Apptainer is installed on Niagara. To verify, run:

  
  which apptainer
  
  A path like /usr/bin/apptainer should be returned.

2. If Apptainer is not enabled, enable it by running:

  
  module load apptainer
  

1.2 Creating the Container

1. Create a definition file to build the container. In your desired directory, run:

  
  cat <<EOF > fmriprep.def
  From: nipreps/fmriprep:latest

  %post
      apt-get update && apt-get install -y python3-pip
      pip3 install pysocks
  EOF
  

2. Build the container with:

  
  apptainer build fmriprep-latest.sif fmriprep-latest.def
  

3. Verify the container by running:

  
  apptainer run fmriprep-latest.sif --version
  
  This should print the version of fMRIPrep in the container.

2. Setting up Templateflow

fMRIPrep will try to download templates from the internet. To avoid this, we set up Templateflow locally.

2.1 Establishing the Templateflow Directory

1. Define the Templateflow home directory, for example:

  
  export TEMPLATEFLOW_HOME=$HOME/templateflow
  

2. Add this to your .bashrc or .zshrc file:

  
  echo "export TEMPLATEFLOW_HOME=$HOME/templateflow" >> ~/.bashrc
  

3. Create the directory if it does not exist:

  
  mkdir -p $TEMPLATEFLOW_HOME
  

2.2 Using Python to Install Templateflow

1. Activate your Python environment. For instructions, see [these instructions](https://docs.scinet.utoronto.ca/index.php/Python#Using_Virtualenv_in_Regular_Python).

2. Install Templateflow:

  
  pip install templateflow
  

3. Verify the TEMPLATEFLOW_HOME environment variable:

  
  echo $TEMPLATEFLOW_HOME
  

4. Start a Python session to install templates:

  
  python
  

  Within Python, run:
  
  import templateflow.api as tf
  tf.templates()
  tf.get(["MNI152NLin2009cAsym", "MNI152NLin6Asym", "OASIS30ANTs", "fsLR", "fsaverage"])
  

5. Exit Python with exit().

3. Establishing an SSH Tunnel to the Niagara Login Node

Establish this tunnel each time you log in to Niagara: ssh -D 44223 nia-login02 -f -N Replace nia-login02 with your current login node (check with echo $HOSTNAME).

4. Preparing a Script to Run fMRIPrep

Assuming you’ve set up everything above, have a BIDS dataset ready, and have a Freesurfer license file, you can use the following example script:

1. !/bin/bash
2. SBATCH --job-name=fmriprep
3. SBATCH --output=specify/where/to/save/fmriprep.log
4. SBATCH --error=specify/where/to/save/fmriprep.err
5. SBATCH --time=12:00:00
6. SBATCH --nodes=1
7. SBATCH --ntasks-per-node=1
8. SBATCH --cpus-per-task=40
9. SBATCH --mail-type=END,FAIL
10. SBATCH --mail-user=your.email@whatever.ca

export APPTAINER_INSTANCE=/path/to/your/container/file export BIDS_DIR=/path/to/your/bids/dataset export FS_LICENSE=/path/to/your/freesurfer/license/file

if [ ! -d $TEMPLATEFLOW_HOME ]; then

   echo "Templateflow directory does not exist: $TEMPLATEFLOW_HOME"
   exit 1

fi

if [ ! -d $BIDS_DIR ]; then

   echo "BIDS directory does not exist: $BIDS_DIR"
   exit 1

fi

if [ ! -d $BIDS_DIR/derivatives/fmriprep ]; then

   mkdir -p $BIDS_DIR/derivatives/fmriprep

fi

if [ ! -f $FS_LICENSE ]; then

   echo "Freesurfer license file does not exist: $FS_LICENSE"
   exit 1

fi

export all_proxy=socks5://localhost:44223 export APPTAINERENV_TEMPLATEFLOW_HOME=/templateflow

apptainer run \

   --cleanenv \
   -B $BIDS_DIR:/data \
   -B $BIDS_DIR/derivatives/fmriprep:/data/derivatives/fmriprep \
   -B $FS_LICENSE:/freesurfer_license.txt \
   -B $TEMPLATEFLOW_HOME:$APPTAINERENV_TEMPLATEFLOW_HOME \
   $APPTAINER_INSTANCE \
   /data \
   /data/derivatives/fmriprep \
   participant \
   --random-seed 42 \
   --omp-nthreads 40 \
   --fs-license-file /freesurfer_license.txt

Make the script executable: chmod +x your_script.sh

5. Running the Script

Submit the script with: sbatch your_script.sh Check the job status with: sq

Good luck!