SciNet Users Documentation - User contributions [en]

Mist

2022-02-01T19:39:04Z

Gauriaur: /* Getting started on Mist */

{{Infobox Computer
|image=[[Image:Mist.jpg|center|300px|thumb]]
|name=Mist
|installed=Dec 2019
|operatingsystem= Red Hat Enterprise Linux 8.2
|loginnode= mist.scinet.utoronto.ca
|nnodes= 54 IBM AC922
|rampernode= 256 GB
|gpuspernode=4 V100-SMX2-32GB
|interconnect=Mellanox EDR
|vendorcompilers= NVCC, IBM XL
|queuetype=Slurm
}}

=Specifications=
Mist is a SciNet-[[#SOSCIP Users |SOSCIP]] joint GPU cluster consisting of 54 IBM AC922 servers. Each node of the cluster has 32 IBM Power9 cores, 256GB RAM and 4 NVIDIA V100-SMX2-32GB GPU with NVLINKs in between. The cluster has InfiniBand EDR interconnection providing GPU-Direct RMDA capability.

'''Important note:''' the majority of computer systems as of 2021 (laptops, desktops, and HPC) use the 64 bit x86 instruction set architecture (ISA) in their microprocessors produced by Intel and AMD. This ISA is incompatible with Mist, whose hardware uses the 64 bit PPC ISA (set to little endian mode). The practical meaning is that x86-compiled binaries (executables and libraries) cannot be installed on Mist. For this reason, the Niagara and Compute Canada software stacks (modules) cannot be made available on Mist, and using closed-source software is only possible when the vendor provides a compatible version of their application. '''Python applications''' almost always rely on bindings to libraries originally written in C or C++, some of them are not available on PyPI or various Conda channels as precompiled binaries compatible with Mist. The recommended way to use Python on Mist is to create a [[#Anaconda (Python)|Conda]] environment and install packages from the anaconda (default) channel, where most popular packages have a linux-ppc64le (Mist-compatible) version available. Some popular machine learning packages should be installed from the internal [[#Open-CE|Open-CE]] channel. Where a compatible Conda package cannot be found, installing from PyPI (<code>pip install</code>) can be attempted. Pip will attempt to compile the package’s source code if no compatible precompiled wheel is available, therefore a compiler module (such as <code>gcc/.core</code>) should be loaded in advance. Some packages require tweaking of the source code or build procedure to successfully compile on Mist, please contact [[#Support|support]] if you need assistance.

= Getting started on Mist =
As of January 22 2022, authentication is only allowed via SSH keys. [https://docs.computecanada.ca/wiki/SSH_Keys Please refer to this page] to generate your SSH key pair and make sure you use them securely.

Mist can be accessed directly:
<pre>
ssh -i /path/to/ssh_private_key -Y MYCCUSERNAME@mist.scinet.utoronto.ca
</pre>
Mist login node '''mist-login01''' can also be accessed via Niagara cluster.
<pre>
ssh -i /path/to/ssh_private_key -Y MYCCUSERNAME@niagara.scinet.utoronto.ca
ssh -Y mist-login01
</pre>
== Storage ==
The filesystem for Mist is shared with Niagara cluster. See [https://docs.scinet.utoronto.ca/index.php/Niagara_Quickstart#Your_various_directories Niagara Storage] for more details.

= Loading software modules =

You have two options for running code on Mist: use existing software, or compile your own. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]] and a list of [[Modules for Mist]] is also available.

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.
== Tips for loading software ==

* We advise '''''against''''' loading modules in your .bashrc. This can lead to very confusing behaviour under certain circumstances. Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>cuda</code> → <code>cuda/11.0.3</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Modules often require other modules to be loaded first. Solve these dependencies by using [[Using_modules#Module_spider | <code>module spider</code>]].

= Available compilers and interpreters =
* <tt>cuda</tt> module has to be loaded first for GPU software.
* For most compiled software, one should use the GNU compilers (<tt>gcc</tt> for C, <tt>g++</tt> for C++, and <tt>gfortran</tt> for Fortran). Loading <tt>gcc</tt> module makes these available.
* The IBM XL compiler suite (<tt>xlc_r, xlc++_r, xlf_r</tt>) is also available, if you load one of the <tt>xl</tt> modules.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> or <tt>spectrum-mpi</tt> module.

=== CUDA ===

The current installed CUDA Tookits are '''11.0.3''' and '''10.2.2 (10.2.89)'''
<pre>
module load cuda/11.0.3
module load cuda/10.2.2
</pre>
*A compiler (GCC, XL or NVHPC/PGI) module must be loaded in order to use CUDA to build any code.
The current NVIDIA driver version is 450.119.04.

===GNU Compilers ===

Available GCC modules are:
<pre>
gcc/9.3.0 (must load CUDA 11)
gcc/8.5.0 (must load CUDA 10 or 11)
gcc/10.3.0 (w/o CUDA)
</pre>

=== IBM XL Compilers ===

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xl/16.1.1.10
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER9 CPU. Information about the IBM XL Compilers can be found at the following links:[https://www.ibm.com/support/knowledgecenter/SSXVZZ_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++],
[https://www.ibm.com/support/knowledgecenter/SSAT4T_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

=== OpenMPI ===
<tt>openmpi/<version></tt> module is avaiable with different compilers including GCC and XL. <tt>spectrum-mpi/<version></tt> module provides IBM Spectrum MPI.

=== NVHPC/PGI ===
PGI compiler is provided in NVHPC (NVIDIA HPC SDK).
<pre>
module load nvhpc/21.3
</pre>

= Software =
== Amber20 ==

Users who hold Amber20 license can build Amber20 from its source code and run on Mist. '''SOSCIP/SciNet doesn't provide Amber license or source code.'''

=== Building Amber20 ===
Modules that are needed for building Amber20:
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 anaconda3/2021.05 cmake/3.19.8
</pre>
Cmake configuration:
<pre>
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/where-amber-install -DCOMPILER=GNU -DMPI=FALSE -DCUDA=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=FALSE -DOPENMP=TRUE -DNCCL=FALSE -DAPPLY_UPDATES=TRUE
</pre>

=== Running Amber20 ===
'''NVIDIA Pascal P100 and later GPUs like V100 do not scale beyond a single GPU'''. It is highly suggested to run Amber20 as a single-gpu job.
A job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP-project-ID>

module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 anaconda3/2021.05
export PATH=$HOME/where-amber-install/bin:$PATH
export LD_LIBRARY_PATH=$HOME/where-amber-install/lib:$LD_LIBRARY_PATH
pmemd.cuda .... <parameters> ...
</pre>

== Anaconda (Python) ==
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation. Anaconda is provided as modules: '''anaconda3'''

To install Anaconda locally, user need to load the module and create a conda environment:
<pre>
module load anaconda3
conda create -n myPythonEnv python=3.8
</pre>
*Note: By default, conda environments are located in '''$HOME/.conda/envs'''. Cache (downloaded tarballs and packages) is under '''$HOME/.conda/pkgs'''. User may run into problem with disk quota if there are too many environments created. To clean conda cache, '''please run: "conda clean -y --all" and "rm -rf $HOME/.conda/pkgs/*" after installation of packages'''.

To activate the conda environment: (should be activated before running python)
<pre>
source activate myPythonEnv
</pre>
Note that you SHOULD NOT use '''conda activate myPythonEnv''' to activate the environment. This leads to all sorts of problems. Once the environment is activated, user can update or install packages via '''conda''' or '''pip'''
<pre>
conda install <package_name> (preferred way to install packages)
pip install <package_name>
</pre>
*Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>
To deactivate:
<pre>
source deactivate
</pre>
To remove a conda environment:
<pre>
conda remove --name myPythonEnv --all
</pre>
To verify that the environment was removed, run:
<pre>
conda info --envs
</pre>

=== Submitting Python Job ===
A single-gpu job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate myPythonEnv
python code.py ...
</pre>

== CuPy ==
[https://cupy.chainer.org CuPy] is an open-source matrix library accelerated with NVIDIA CUDA. It also uses CUDA-related libraries including cuBLAS, cuDNN, cuRand, cuSolver, cuSPARSE, cuFFT and NCCL to make full use of the GPU architecture. CuPy is an implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it. It supports a subset of numpy.ndarray interface.

CuPy can be install into any conda environment. Python packages: numpy, six and fastrlock are required. cuDNN and NCCL are optional.
<pre>
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 nccl/2.9.9 anaconda3/2021.05
conda create -n cupy-env python=3.8 numpy six fastrlock
source activate cupy-env
CFLAGS="-I$MODULE_CUDNN_PREFIX/include -I$MODULE_NCCL_PREFIX/include -I$MODULE_CUDA_PREFIX/include" LDFLAGS="-L$MODULE_CUDNN_PREFIX/lib64 -L$MODULE_NCCL_PREFIX/lib" CUDA_PATH=$MODULE_CUDA_PREFIX pip install cupy
#building/installing CuPy will take a few minutes
</pre>

== Gromacs ==
[http://www.gromacs.org/ GROMACS] is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.
*'''GROMACS 2019'''
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2019.6
</pre>
*'''GROMACS 2020 and 2021''' Thread-MPI version supports full GPU enablement of all key computational sections. The GPU is used throughout the timestep and repeated CPU-GPU transfers are eliminated. Users are suggested to carefully verify the results.
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2020.4
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2020.6
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.2
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 openmpi/4.1.1+ucx-1.10.0 gromacs/2021.2 (testing purpose only)
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.4
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.5
</pre>
=== Small/Medium Simulation ===
Due to the lack of PME domain decomposition support on GPU, Gromacs uses CPU to calculate PME when using multiple GPUs. '''It is always recommended to use a single GPU to do small and medium sized simulations with Gromacs.''' By using only 1 tMPI thread (w/ multiple OpenMP threads) on a single GPU, both non-bonded PP and PME are atomically offloaded to GPU when possible.
* Gromacs 2019 example:
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1

module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2019.6
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
gmx mdrun -pin off -ntmpi 1 -ntomp 8 ... <other parameters>
</pre>

* Gromacs 2020 or 2021 example:
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.5
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
export GMX_FORCE_UPDATE_DEFAULT_GPU=true
gmx mdrun -pin off -ntmpi 1 -ntomp 8 ... <other parameters>
</pre>

=== Large Simulation ===
If memory size (~58GB) for single-gpu job is not sufficient for the simulation, multiple GPUs can be used. It is suggested to test starting with one full node with 4GPUs and force PME on GPU. Multiple PME ranks are not supported with PME on GPU, so if GPU is used for the PME calculation -npme (number of PME ranks) must be set to 1. If PME has less work than PP, it is suggested to run multiple ranks per GPU, so the GPU for PME rank can also do some work on PP rank(s).
'''If your simulation can fit in a single GPU job, please use single GPU to get much higher efficiency. Do not waste 3 additional GPU resource for getting only a small performance improvement.
'''
*An example using 4 GPUs, 7 PP ranks/tmpi threads + 1 PME rank/tmpi thread: ('''-pin on -pme gpu -npme 1''' must be added to mdrun command in order to force GPU to do PME)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.5

export OMP_NUM_THREADS=4
gmx mdrun -ntmpi 8 -pin on -pme gpu -npme 1 ... <add your parameters>
</pre>
*It is suggested to also test using '''-ntmpi 4''' and '''export OMP_NUM_THREADS=8''' if you receive a NOTE in Gromacs output saying "% performance was lost because the PME ranks had more work to do than the PP ranks". In this case, NVIDIA MPS is not needed since there is only one MPI rank per GPU.
*'''Please note that the solving of PME on GPU is still only the initial version supporting this behaviour, and comes with a set of limitations outlined further below.'''
<pre>
* Only a PME order of 4 is supported on GPUs.
* PME will run on a GPU only when exactly one rank has a PME task, ie. decompositions with multiple ranks doing PME are not supported.
* Only single precision is supported.
* Free energy calculations where charges are perturbed are not supported, because only single PME grids can be calculated.
* Only dynamical integrators are supported (ie. leap-frog, Velocity Verlet, stochastic dynamics)
* LJ PME is not supported on GPUs.
</pre>
*An example using 4 GPUs, '''PME on CPU''': ('''-pin on''' must be added to mdrun command for proper CPU thread bindings)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.5

export OMP_NUM_THREADS=4
gmx mdrun -ntmpi 8 -pin on ... <add your parameters>

# "-ntmpi 16, OMP_NUM_THREADS=2" and "-ntmpi 4, OMP_NUM_THREADS=8" should also be tested.
# num_thread_MPI_ranks(-ntmpi) * num_OpenMP_threads = 32
</pre>
*'''If your simulation can fit in a single GPU job, please use single GPU to get much higher efficiency. Do not waste 3 additional GPU resource for getting only a small performance improvement.'''
*'''NOTE: The above examples will NOT work with multiple nodes. If simulation is too large for a single GPU node, please contact SciNet/SOSCIP support.'''

== NAMD ==
[http://www.ks.uiuc.edu/Research/namd/ NAMD] is a parallel, object-oriented molecular dynamics code designed for high-performance simulation of large biomolecular systems.
=== 2.14 ===
<pre>
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
</pre>
==== Running with single GPU ====
If you have many jobs to run, it is always suggested to run with a single gpu per job. This makes jobs easier to be scheduled and gives better overall performance.
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=1
#SBATCH --nodes=1

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -bind-to none -hostfile nodelist-$SLURM_JOB_ID `which namd2` +idlepoll +ppn 8 +p 8 stmv.namd
</pre>

==== Running with one process per node (4 GPUs)====
An example of the job script (using 1 node, '''one process per node''', 32 CPU threads per process + 4 GPUs per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 32 +p $((32*SLURM_NTASKS)) stmv.namd
</pre>
==== Running with one process per GPU (4 GPUs)====
NAMD may scale better if using '''one process per GPU'''. Please do your own benchmark.
An example of the job script (using 1 node, '''one process per GPU''', 8 CPU threads per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 4 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 8 +p $((8*SLURM_NTASKS)) stmv.namd
</pre>

== Open-CE ==
[https://github.com/open-ce/open-ce Open-CE] is an '''IBM''' repo for feedstock collection, environment data, and scripts for building Tensorflow, Pytorch, and other machine learning packages and dependencies. Open-CE is distributed as a '''conda channel''' on Mist cluster.
'''Available packages and versions are listed here [https://github.com/open-ce/open-ce/releases/tag/open-ce-v1.5.1 Open-CE Releases]'''. Currently only python 3.8 and CUDA 11.2 are supported. If you need a different python or cuda version, please contact SOSCIP or SciNet support.

*Packages can be installed by setting Open-CE conda channel:
<pre>
conda install -c /scinet/mist/ibm/open-ce python=3.8 cudatoolkit=11.2 PACKAGE
</pre>
*Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

{| class="wikitable"
|+Available Packages:
|-
|Tensorflow
|TensorFlow Estimators
|TensorFlow Probability
|TensorBoard
|TensorBoard Data Server
|TensorFlow Text
|TensorFlow Model Optimizations
|TensorFlow Addons
|TensorFlow Datasets
|TensorFlow Hub
|-
|TensorFlow MetaData
|PyTorch
|TorchText
|TorchVision
|PyTorch Lightning
|PyTorch Lightning Bolts
|ONNX
|Onnx-runtime
|skl2onnx
|tf2onnx
|-
|onnxmltools
|onnxconverter-common
|XGBoost
|LightGBM
|Transformers
|Tokenizers
|SentencePiece
|Spacy
|DALI
|OpenCV
|-
|Horovod
|PyArrow
|grpc
|uwsgi
|ORC
|}

== PyTorch ==
=== Installing from IBM Open-CE Conda Channel ===
The easiest way to install PyTorch on Mist is using IBM's Conda channel. User needs to prepare a conda environment and install PyTorch using IBM's Open-CE Conda channel.
<pre>
module load anaconda3
conda create -n pytorch_env python=3.8
source activate pytorch_env
conda install -c /scinet/mist/ibm/open-ce pytorch=1.10.1 cudatoolkit=11.2
or
conda install -c /scinet/mist/ibm/open-ce-1.2 pytorch=1.7.1 cudatoolkit=11.0 (or 10.2)
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>
Add below command into your job script before python command to get deterministic results, see details here: [https://github.com/pytorch/pytorch/issues/39849]
<pre>
export CUBLAS_WORKSPACE_CONFIG=:4096:2
</pre>

== RAPIDS ==
The [https://rapids.ai RAPIDS] is a suite of open source software libraries that gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. The RAPIDS data science framework includes a collection of libraries: '''cuDF(GPU DataFrames)''', '''cuML(GPU Machine Learning Algorithms)''', '''cuStrings(GPU String Manipulation)''', etc.

=== Installing from IBM Conda Channel ===
The easiest way to install RAPIDS on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install powerai-rapids using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n rapids_env python=3.7
source activate rapids_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/ powerai-rapids
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== TensorFlow and Keras ==
=== Installing from IBM Conda Channel ===
The easiest way to install TensorFlow and Keras on Mist is using IBM's Open-CE Conda channel. User needs to prepare a conda environment and install TensorFlow using IBM's Open-CE Conda channel.
<pre>
module load anaconda3
conda create -n tf_env python=3.8
source activate tf_env
conda install -c /scinet/mist/ibm/open-ce tensorflow==2.7.0 cudatoolkit=11.2
or
conda install -c /scinet/mist/ibm/open-ce-1.2 tensorflow==2.4.3 cudatoolkit=11.0 (or 10.2)
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

= Testing and debugging =
You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.
* Small test jobs can be run on the login node. Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than one gpu and a few cores.

* Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
mist-login01:~$ debugjob --clean -g G
where G is the number of gpus, If G=1, this gives an interactive session for 2 hours, whereas G=4 gets you a single node with 4 gpus for 30 minutes, and with G=8 (the maximum) gets you 2 nodes each with 4 gpus for 15 minutes. The <tt>--clean</tt> argument is optional but recommended as it will start the session without any modules loaded, thus mimicking more closely what happens when you submit a job script. Users needs to load module and activate the conda environment after a debug job starts. It is recommended to do a 'conda clean' before 'source activate ENV' in a debug job if --clean flag is missed.

= Submitting jobs =
Once you have compiled and tested your code or workflow on the Mist login nodes, and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. Your jobs will run on some of Mist's 53 compute nodes. When and where your job runs is determined by the scheduler.

Mist uses SLURM as its job scheduler. It is configured to allow only '''Single-GPU jobs''' and '''Full-node jobs (4 GPUs per node)'''.

You submit jobs from a login node by passing a script to the sbatch command:

mist-login01:scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course. In most cases, you should not submit from your $HOME directory, but rather, from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

Example job scripts can be found below.
Keep in mind:
* Scheduling is by single gpu or by full node, so you ask only 1 gpu or 4 gpus per node.
* Your job's maximum walltime is 24 hours.
* Jobs must write their output to your scratch or project directory (home is read-only on compute nodes).
* Compute nodes have no internet access.
* Your job script will not remember the modules you have loaded, so it needs to contain "module load" commands of all the required modules (see examples below).
== SOSCIP Users ==
*[https://www.soscip.org SOSCIP] is a consortium to bring together industrial partners and academic researchers and provide them with sophisticated advanced computing technologies and expertise to solve social, technical and business challenges across sectors and drive economic growth.

If you are working on a SOSCIP project, please contact [mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca] to have your user account added to SOSCIP project accounts. SOSCIP users need to submit jobs with additional SLURM flag to get higher priority:
<pre>
#SBATCH -A soscip-<SOSCIP_PROJECT_ID> #e.g. soscip-3-001
OR
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID>
</pre>

== Single-GPU job script ==
For a single GPU job, each will have a quarter of the node which is 1 GPU + 8/32 CPU Cores/Threads + ~58GB CPU memory. '''Users should never ask CPU or Memory explicitly.''' If running MPI program, user can set --ntasks to be the number of MPI ranks. '''Do NOT set --ntasks for non-MPI programs.'''
*It is suggested to use NVIDIA Multi-Process Service (MPS) if running multiple MPI ranks on one GPU.
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate conda_env
python code.py ...
</pre>

== Full-node job script ==
'''If you are not sure the program can be executed on multiple GPUs, please follow the single-gpu job instruction above or contact SciNet/SOSCIP support.'''

Multi-GPU job should ask for a minimum of one full node (4 GPUs). User need to specify "compute_full_node" partition in order to get all resource on a node.
*An example for a 1-node job:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4 #this only affects MPI job
#SBATCH --time=1:00:00
#SBATCH -p compute_full_node
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load <modules you need>
Run your program
</pre>

= Jupyter Notebooks =
SciNet’s [[Jupyter Hub]] is a Niagara-type node; it has a different CPU architecture and no GPUs. Conda environments prepared on Mist will not work there properly. Users who need to use Jupyter Notebook to develop and test some aspects of their workflow can create their own server on the Mist login node and use an SSH tunnel to connect to it from outside. Users who choose to do so have to keep in mind that the login node is a shared resource, and heavy calculations should be done only on compute nodes. Processes (including iPython kernels used by the notebooks) are limited to one hour of total CPU time: idle time will not be counted toward this one hour, and use of multiple cores will count proportionally to the number of cores (i.e. a kernel using all 128 virtual cores on the node will be killed after 28 seconds). Idle notebooks can still burden the node by hogging system and GPU memory, please be mindful of other users and terminate notebooks when work is done.

As an example, let us create a new Conda environment and activate it:
<pre>
module load anaconda3
conda create -n jupyter_env python=3.7
source activate jupyter_env
</pre>
Install the Jupyter Notebook server:
<pre>
conda install notebook
</pre>

== Running the notebook server ==
When the Conda environment is active, enter:
<pre>
jupyter-notebook
</pre>
By default, the Jupyter Notebook server uses port 8888 (can be overridden with the <code>--port</code> option). If another user has already started their own server, the default port may be busy, in which case the server will be listening on a different port. Once launched, the server will output some information to the terminal that will include the actual port number used and a 48-character token. For example:
<pre>http://localhost:8890/?token=54c4090d……</pre>
In this example, the server is listening on port 8890.

== Creating a tunnel ==
In order to access this port remotely (i.e. from your office or home), an [https://en.wikipedia.org/wiki/Tunneling_protocol#Secure_Shell_tunneling SSH tunnel] has to be established. Please refer to your SSH client’s documentation for instructions on how to do that. For the OpenSSH client (standard in most Linux distributions and macOS), a tunnel can be opened in a separate terminal session to the one where the Jupyter Notebook server is running. In the new terminal, issue this command:
<pre>
ssh -L8888:localhost:8890 <username>@mist.scinet.utoronto.ca
</pre>
(replace <code><username></code> with your actual username) The tunnel is open as long as this SSH connection is alive. In this example, we tunnel Mist login node’s port 8890 (where our server is assumed to be running) to our home computer’s port 8888 (any other free port is fine). The notebook can be accessed in the browser at the <code><nowiki>http://localhost:8888</nowiki></code> address (followed by <code>/?token=54c4090d……</code>, or the token can be input on the webpage).

== Using Jupyter on compute nodes ==

You can use the instructions here to set up a Jupyter Notebook server on a compute node (including a [[#Testing_and_debugging|debugjob]]). '''We strongly discourage''' you from running an interactive notebook on a compute node (other than for a debugjob), scheduled jobs run in arbitrary times and are not meant to be interactive. Jupyter notebooks can be run non-interactively or converted to Python scripts.

To launch the Jupyter Notebook server, load the <code>anaconda3</code> module and activate your environment as before (by adding the appropriate lines to the submission script, if you are not using the compute node with an interactive shell). Launching the server has to be done like so:
<pre>
HOME=/dev/shm/$USER jupyter-notebook
</pre>
That is because Jupyter will fail unless it can write to the home folder, which is read-only from compute nodes. This modification of the <code>$HOME</code> environment variable will carry over into the notebooks, which is usually not a problem, but in case the notebook relies on this environment variable (e.g. to read certain files), it can be reset manually in the notebook (<code>import os; os.environ['HOME']=……</code>).

Because compute nodes are not accessible from the Internet, tunneling has to be done twice, once from the remote location (office or home) to the Mist login node, and then from the login node to the compute node. Assuming the server is running on port 8890 of the mist006 node, open the first tunnel in a new terminal session in the remote computer:
<pre>
ssh -L8888:localhost:9999 <username>@mist.scinet.utoronto.ca
</pre>
where 9999 is any available port on the Mist login node (to test port availability enter <code>ss -Hln src :9999</code> in the terminal when connected to the Mist login node; an empty output indicates that the port is free). In the same session in the login node that was created with the above command, open the second tunnel to the compute node:
<pre>
ssh -L9999:localhost:8890 mist006
</pre>
Be aware that the second tunnel will automatically disconnect once the job on the compute node times out or is relinquished. The Jupyter Notebook server running on the compute node can now be accessed from the browser as in the previous subsection.

= Support =

SciNet inquiries:
* [mailto:support@scinet.utoronto.ca support@scinet.utoronto.ca]

SOSCIP inquiries:
*[mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca]

Niagara Quickstart

2022-02-01T19:35:48Z

Gauriaur: /* Logging in */

{{Infobox Computer
|image=[[Image:Niagara.jpg|center|300px|thumb]]
|name=Niagara
|installed=Jan 2018/March 2020
|operatingsystem= CentOS 7.6
|loginnode= niagara.scinet.utoronto.ca
|nnodes= 2,024 nodes (80,960 cores)
|rampernode=188 GiB / 202 GB
|corespernode=40 (80 hyperthreads)
|interconnect=Mellanox Dragonfly+
|vendorcompilers= icc (C) ifort (fortran) icpc (C++)
|queuetype=Slurm
}}

=Specifications=

The Niagara cluster is a large cluster of 2,024 Lenovo SD530 servers each with 40 Intel "Skylake" at 2.4 GHz or 40 Intel "CascadeLake" cores at 2.5 GHz.
The peak performance of the cluster is about 3.6 PFlops (6.25 PFlops theoretical). It was the 53rd fastest supercomputer on the [https://www.top500.org/list/2018/06/?page=1 TOP500 list of June 2018], and is at number 113 on the [https://www.top500.org/lists/top500/list/2021/06/ current list (June 2021)].

Each node of the cluster has 188 GiB / 202 GB RAM per node (at least 4 GiB/core for user jobs). Being designed for large parallel workloads, it has a fast interconnect consisting of EDR InfiniBand in a Dragonfly+ topology with Adaptive Routing. The compute nodes are accessed through a queueing system that allows jobs with a minimum of 15 minutes and a maximum of 24 hours and favours large jobs.

* See the [https://www.youtube.com/watch?v=l-E2CFGh0BE&feature=youtu.be "Intro to Niagara"] recording

More detailed hardware characteristics of the Niagara supercomputer can be found [https://docs.computecanada.ca/wiki/Niagara on this page].

Note: Documentation about the "GPU expansion to Niagara" called "Mist" can be found on [[Mist | its own page]].

= Getting started on Niagara =

Access to Niagara is not enabled automatically for everyone with a Compute Canada account, but anyone with an active Compute Canada account can get their access enabled.

If you have an active Compute Canada account but you do not have access to Niagara yet (e.g. because you are new to SciNet or belong to a group whose primary PI does not have an allocation as granted in the annual [https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions Compute Canada RAC]), go to the [https://ccdb.computecanada.ca/services/opt_in opt-in page on the CCDB site]. After clicking the "Join" button, it usually takes only one or two business days for access to be granted.

Please read this document carefully. The [https://docs.scinet.utoronto.ca/index.php/FAQ FAQ] is also a useful resource. If at any time you require assistance, or if something is unclear, please do not hesitate to [mailto:support@scinet.utoronto.ca contact us].

== Logging in ==

Niagara runs CentOS 7, which is a type of Linux. You will need to be familiar with Linux systems to work on Niagara. If you are not it will be worth your time to review our [https://support.scinet.utoronto.ca/education/browse.php?category=-1&search=scmp101&include=all&filter=Filter Introduction to Linux Shell] class.

As with all SciNet and CC (Compute Canada) compute systems, access to Niagara is done via [[SSH]] (secure shell) only. As of January 22 2022, authentication is only allowed via SSH keys. [https://docs.computecanada.ca/wiki/SSH_Keys Please refer to this page] to generate your SSH key pair and make sure you use them securely.

Open a terminal window (e.g. Connecting with [https://docs.computecanada.ca/wiki/Connecting_with_PuTTY PuTTY] on Windows or Connecting with [https://docs.computecanada.ca/wiki/Connecting_with_MobaXTerm MobaXTerm]), then SSH into the Niagara login nodes with your CC credentials:

$ ssh -i /path/to/ssh_private_key -Y MYCCUSERNAME@niagara.scinet.utoronto.ca

or

$ ssh -i /path/to/ssh_private_key -Y MYCCUSERNAME@niagara.computecanada.ca

The first time you login to Niagara, please make sure you are actually accessing Niagara by checking if the login node ssh host key fingerprint matches [[SSH_Changes_in_May_2019 | (See here how)]]. This check prevents you from falling victim of [https://en.wikipedia.org/wiki/Man-in-the-middle_attack man-in-the-middle attacks.]

* The Niagara login nodes are where you develop, edit, compile, prepare and submit jobs.
* These login nodes are not part of the Niagara compute cluster, but have the same architecture, operating system, and software stack.
* The optional <code>-Y</code> is needed to open windows from the Niagara command-line onto your local X server.
* You can only connect 4 times in a 2-minute window to the login nodes.
* To run on Niagara's compute nodes, you must [[#Submitting_jobs | submit a batch job]].

If you cannot log in, be sure to first check the [https://docs.scinet.utoronto.ca System Status] on this site's front page.

== Your various directories ==

By virtue of your access to Niagara you are granted storage space on the system. There are several directories available to you, each indicated by an associated environment variable.

=== home and scratch ===

You have a home and scratch directory on the system, the paths to which are stored in the environment variables $HOME and $SCRATCH. The locations are of the form

$HOME=/home/g/groupname/myccusername
$SCRATCH=/scratch/g/groupname/myccusername

where groupname is the name of your PI's group, and myccusername is your CC username. For example:

nia-login07:~$ pwd
/home/s/scinet/rzon
nia-login07:~$ cd $SCRATCH
nia-login07:rzon$ pwd
/scratch/s/scinet/rzon

NOTE: home is read-only on compute nodes.

=== project and archive/nearline ===

Users from groups with [https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions RAC storage allocation] will also have a project directory and possible an archive (a.k.a. "nearline") directory, the paths to which are stored in the environment variables $PROJECT and $ARCHIVE. They follow the naming convention:

$PROJECT=/project/g/groupname/myccusername
$ARCHIVE=/archive/g/groupname/myccusername

NOTE: Currently archive space is available only via [[HPSS]], and is not accessible on the Niagara login, compute, or datamover nodes.

'''''IMPORTANT: Future-proof your scripts'''''

When writing your scripts, use the environment variables (<tt>$HOME</tt>, <tt>$SCRATCH</tt>, <tt>$PROJECT</tt>, <tt>$ARCHIVE</tt>) instead of the actual paths! The paths may change in the future.

=== Storage and quotas ===

You should familiarize yourself with the [[Data_Management#Purpose_of_each_file_system | various file systems]], what purpose they serve, and how to properly use them. This table summarizes the various file systems. See the [[Data_Management | Data Management]] page for more details.

{| class="wikitable"
! location
!colspan="2"| quota
!align="right"| block size
! expiration time
! backed up
! on login nodes
! on compute nodes
|-
| $HOME
|colspan="2"| 100 GB / 250,000 files per user
|align="right"| 1 MB
|
| yes
| yes
| read-only
|-
|rowspan="2"| $SCRATCH
|colspan="2"| 25 TB / 6,000,000 file per user
|align="right" rowspan="2" | 16 MB
|rowspan="2"| 2 months
|rowspan="2"| no
|rowspan="2"| yes
|rowspan="2"| yes
|-
|align="right"|50-500TB per group
|align="right"|[[Data_Management#Quotas_and_purging | depending on group size]]
|-
| $PROJECT
|colspan="2"| by group allocation
|align="right"| 16 MB
|
| yes
| yes
| yes
|-
| $ARCHIVE
|colspan="2"| by group (nearline) allocation
|align="right"|
|
| dual-copy
| no
| no
|-
| $BBUFFER
|colspan="2"| 10 TB per user
|align="right"| 1 MB
| very short
| no
| yes
| yes
|}

=== Moving data to Niagara ===

If you need to move data to Niagara for analysis, or when you need to move data off of Niagara, use the following guidelines:
* If your data is less than 10GB, move the data using the login nodes.
* If your data is greater than 10GB, move the data using the datamover nodes nia-datamover1.scinet.utoronto.ca and nia-datamover2.scinet.utoronto.ca .

Details of how to use the datamover nodes can be found on the [[Data_Management#Moving_data | Data Management ]] page.

= Loading software modules =

You have two options for running code on Niagara: use existing software, or [[Niagara_Quickstart#Compiling_on_Niagara:_Example | compile your own]]. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]].

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.
== Software stacks: NiaEnv and CCEnv ==

On Niagara, there are two available software stacks:

=== NiaEnv ===

A [https://docs.scinet.utoronto.ca/index.php/Modules_specific_to_Niagara Niagara software stack] tuned and compiled for this machine. This stack is available by default, but if not, can be reloaded with
<pre>module load NiaEnv</pre>
This loads the default (set of modules), which is currently the 2019b epoch. Before September 1, the default was NiaEnv/2018a. Users are encourage to use the 2019b stack, but to make sure old job scripts or older software installations in your home directory continue to work, you may need to use
<pre>module load NiaEnv/2018a</pre>
You can override the system default for the epoch version by creating a file called <tt>.modulerc</tt> in your home directory with the line <tt>module-version NiaEnv/VERSION default</tt>, e.g. like so:
<pre>
echo "module-version NiaEnv/2019b default" > $HOME/.modulerc
</pre>
After this, subsequent logins and jobs will use the 2019b stack even when the system default is different.
Similarly, you can make an older epoch your personal default, like so
<pre>
echo "module-version NiaEnv/2018a default" > $HOME/.modulerc
</pre>

No modules are loaded by default on Niagara except NiaEnv.

=== CCEnv ===

The same [https://docs.computecanada.ca/wiki/Modules software stack available on Compute Canada's General Purpose clusters] [https://docs.computecanada.ca/wiki/Graham Graham] and [https://docs.computecanada.ca/wiki/Cedar Cedar] can be used on Niagara too, with:
<pre>module load CCEnv</pre>
Or, if you want the same default modules loaded as on Béluga, then do
<pre>module load CCEnv StdEnv</pre>
or, if you want the same default modules loaded as on Cedar and Graham, do
<pre>module load CCEnv arch/avx2 StdEnv</pre>

== Tips for loading software ==

* We advise '''''against''''' loading modules in your .bashrc. This can lead to very confusing behaviour under certain circumstances. Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>intel</code> → <code>intel/2018.2</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Modules often require other modules to be loaded first. Solve these dependencies by using [[Using_modules#Module_spider | <code>module spider</code>]].

= Available compilers and interpreters =

* For most compiled software, one should use the Intel compilers (<tt>icc</tt> for C, <tt>icpc</tt> for C++, and <tt>ifort</tt> for Fortran). Loading an <tt>intel</tt> module makes these available.
* The GNU compiler suite (<tt>gcc, g++, gfortran</tt>) is also available, if you load one of the <tt>gcc</tt> modules.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> or <tt>intelmpi</tt> module.
* Open source interpreted, interactive software is also available:
** [[Python]]
** [[R]]
** Julia
** [[Octave]]

Please visit the corresponding page for details on using these tools. For information on running MATLAB applications on Niagara, visit [[MATLAB| this page]].

= Using Commercial Software =

May I use commercial software on Niagara?
* Possibly, but you have to bring your own license for it. You can connect to an external license server using [[SSH_Tunneling | ssh tunneling]].
* SciNet and Compute Canada have an extremely large and broad user base of thousands of users, so we cannot provide licenses for everyone's favorite software.
* Thus, the only freely available commercial software installed on Niagara is software that can benefit everyone: Compilers, math libraries and debuggers.
* That means no [[MATLAB]], Gaussian, IDL,
* Open source alternatives like Octave, [[Python]], and [[R]] are available.
* We are happy to help you to install commercial software for which you have a license.
* In some cases, if you have a license, you can use software in the Compute Canada stack.
The list of commercial software which is installed on Niagara, for which you will need a license to use, can be found on the [[Commercial_software | commercial software page]].

= Compiling on Niagara: Example =

Suppose one wants to compile an application from two c source files, appl.c and module.c, which use the Math Kernel Library. This is an example of how this would be done:
<source lang="bash">
nia-login07:~$ module load NiaEnv/2019b
nia-login07:~$ module list
Currently Loaded Modules:
1) NiaEnv/2019b (S)
Where:
S: Module is Sticky, requires --force to unload or purge

nia-login07:~$ module load intel/2019u4

nia-login07:~$ ls
appl.c module.c

nia-login07:~$ icc -c -O3 -xHost -o appl.o appl.c
nia-login07:~$ icc -c -O3 -xHost -o module.o module.c
nia-login07:~$ icc -o appl module.o appl.o -mkl

nia-login07:~$ ./appl
</source>
Note:
* The optimization flags -O3 -xHost allow the Intel compiler to use instructions specific to the architecture CPU that is present (instead of for more generic x86_64 CPUs).
* Linking with the Intel Math Kernel Library (MKL) is easy when using the intel compiler, it just requires the -mkl flags.
* If compiling with gcc, the optimization flags would be -O3 -march=native. For the way to link with the MKL, it is suggested to use the [https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor MKL link line advisor].

= Testing and Debugging =

You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.
* Small test jobs can be run on the login nodes. Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than a couple of cores.
* You can run the [[Parallel Debugging with DDT|DDT]] debugger on the login nodes after <code>module load ddt</code>.
* Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
nia-login07:~$ debugjob --clean N
where N is the number of nodes, If N=1, this gives an interactive session one 1 hour, when N=4 (the maximum), it gives you 22 minutes. The <tt>--clean</tt> argument is optional but recommended as it will start the session without any modules loaded, thus mimicking more closely what happens when you submit a job script.

Finally, if your debugjob process takes more than 1 hour, you can request an interactive job from the regular queue using the salloc command. Note, however, that this may take some time to run, since it will be part of the regular queue, and will be run when the scheduler decides.
nia-login07:~$ salloc --nodes N --time=M:00:00 --x11
where N is again the number of nodes, and M is the number of hours you wish the job to run.
The <tt>--x11</tt> is required if you need to use graphics while testing your code through salloc, e.g. when using a debugger such as [[Parallel Debugging with DDT|DDT]] or DDD, See the [[Testing_With_Graphics | Testing with graphics]] page for the options in that case.

= Submitting jobs =




Once you have compiled and tested your code or workflow on the Niagara login nodes, and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. Your jobs will run on some of Niagara's 1548 compute nodes. When and where your job runs is determined by the scheduler.

Niagara uses SLURM as its job scheduler. More-advanced details of how to interact with the scheduler can be found on the [[Slurm | Slurm page]].

You submit jobs from a login node by passing a script to the sbatch command:

nia-login07:scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course. Note that you must submit your job from a login node. You cannot submit jobs from the datamover nodes.

In most cases, you should not submit from your $HOME directory, but rather, from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

Jobs will run under your group's RRG allocation, or, if the your group has none, under a RAS allocation (previously called `default' allocation).

Some example job scripts can be found below.

Keep in mind:
* Scheduling is by node, so in multiples of 40 cores.
* Your job's maximum walltime is 24 hours.
* Jobs must write their output to your scratch or project directory (home is read-only on compute nodes).
* Compute nodes have no internet access.
* Your job script will not remember the modules you have loaded, so it needs to contain "module load" commands of all the required modules (see examples below).
* [[Data_Management#Moving_data | Move your data]] to Niagara before you submit your job.

== Scheduling by Node ==

On many systems that use SLURM, the scheduler will deduce from the specifications of the number of tasks and the number of cpus-per-node what resources should be allocated. On Niagara things are a bit different.
* All job resource requests on Niagara are scheduled as a multiple of '''nodes'''.
* The nodes that your jobs run on are exclusively yours, for as long as the job is running on them.
** No other users are running anything on them.
** You can [[SSH]] into them to see how things are going.
* Whatever your requests to the scheduler, it will always be translated into a multiple of nodes allocated to your job.
* Memory requests to the scheduler are of no use. Your job always gets N x 202GB of RAM, where N is the number of nodes and 202GB is the amount of memory on the node.
* If you run serial jobs you must still use all 40 cores on the node. Visit the [[Running_Serial_Jobs_on_Niagara | serial jobs]] page for examples of how to do this.
* Since there are 40 cores per node, your job should use N x 40 cores. If you do not, we will contact you to help you optimize your workflow. Or you can [mailto:support@scinet.utoronto.ca contact us] to get assistance.

== Limits ==

There are limits to the size and duration of your jobs, the number of jobs you can run and the number of jobs you can have queued. It matters whether a user is part of a group with a [https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions/ Resources for Research Group allocation] or not. It also matters in which 'partition' the job runs. 'Partitions' are SLURM-speak for use cases. You specify the partition with the <tt>-p</tt> parameter to <tt>sbatch</tt> or <tt>salloc</tt>, but if you do not specify one, your job will run in the <tt>compute</tt> partition, which is the most common case.

{| class="wikitable"
!Usage
!Partition
!Limit on Running jobs
!Limit on Submitted jobs (incl. running)
!Min. size of jobs
!Max. size of jobs
!Min. walltime
!Max. walltime
|-
|Compute jobs ||compute || 50 || 1000 || 1 node (40 cores) || default: 20 nodes (800 cores) with allocation: 1000 nodes (40000 cores)|| 15 minutes || 24 hours
|-
|Testing or troubleshooting || debug || 1 || 1 || 1 node (40 cores) || 4 nodes (160 cores)|| N/A || 1 hour
|-
|Archiving or retrieving data in [[HPSS]]|| archivelong || 2 per user (5 in total) || 10 per user || N/A || N/A|| 15 minutes || 72 hours
|-
|Inspecting archived data, small archival actions in [[HPSS]] || archiveshort vfsshort || 2 per user|| 10 per user || N/A || N/A || 15 minutes || 1 hour
|}

Even if you respect these limits, your jobs will still have to wait in the queue. The waiting time depends on many factors such as your group's allocation amount, how much allocation has been used in the recent past, the number of requested nodes and walltime, and how many other jobs are waiting in the queue.

== File Input/Output Tips ==

It is important to understand the file systems, so as to perform your file I/O (Input/Output) responsibly. Refer to the [[Data_Management | Data Management]] page for details about the file systems.
* Your files can be seen on all Niagara login and compute nodes.
* $HOME, $SCRATCH, and $PROJECT all use the parallel file system called GPFS.
* GPFS is a high-performance file system which provides rapid reads and writes to large data sets in parallel from many nodes.
* Accessing data sets which consist of many, small files leads to poor performance on GPFS.
* Avoid reading and writing lots of small amounts of data to disk. Many small files on the system waste space and are slower to access, read and write. If you must write many small files, use [[User_Ramdisk | ramdisk]].
* Write data out in a binary format. This is faster and takes less space.
* The [[Burst Buffer]] is another option for I/O heavy-jobs and for speeding up [[Checkpoints|checkpoints]].

== Example submission script (MPI) ==

<source lang="bash">#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=1:00:00
#SBATCH --job-name=mpi_job
#SBATCH --output=mpi_output_%j.txt
#SBATCH --mail-type=FAIL

cd $SLURM_SUBMIT_DIR

module load NiaEnv/2019b
module load intel/2019u4
module load openmpi/4.0.1

mpirun ./mpi_example
# or "srun ./mpi_example"
</source>
Submit this script from your scratch directory with the command:

nia-login07:scratch$ sbatch mpi_job.sh

<ul>
<li>First line indicates that this is a bash script.</li>
<li>Lines starting with <code>#SBATCH</code> go to SLURM.</li>
<li>sbatch reads these lines as a job request (which it gives the name <code>mpi_job</code>)</li>
<li>In this case, SLURM looks for 2 nodes each running 40 tasks (for a total of 80 tasks), for 1 hour</li>
<li>Note that the mpifun flag "--ppn" (processors per node) is ignored.</li>
<li>Once it found such a node, it runs the script:
<ul>
<li>Change to the submission directory;</li>
<li>Loads modules;</li>
<li>Runs the <code>mpi_example</code> application (SLURM will inform mpirun or srun on how many processes to run).
</li>
</ul>
<li>To use hyperthreading, just change <code>--ntasks-per-node=40</code> to <code>--ntasks-per-node=80</code>, and add <code>--bind-to none</code> to the mpirun command (the latter is necessary for OpenMPI only, not when using IntelMPI).</li>
</ul>

== Example submission script (OpenMP) ==

<source lang="bash">#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=40
#SBATCH --time=1:00:00
#SBATCH --job-name=openmp_job
#SBATCH --output=openmp_output_%j.txt
#SBATCH --mail-type=FAIL

cd $SLURM_SUBMIT_DIR

module load NiaEnv/2019b
module load intel/2019u4

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

./openmp_example
# or "srun ./openmp_example".
</source>
Submit this script from your scratch directory with the command:

nia-login07:~$ sbatch openmp_job.sh

* First line indicates that this is a bash script.
* Lines starting with <code>#SBATCH</code> go to SLURM.
* sbatch reads these lines as a job request (which it gives the name <code>openmp_job</code>) .
* In this case, SLURM looks for one node with 40 cores to be run inside one task, for 1 hour.
* Once it found such a node, it runs the script:
** Change to the submission directory;
** Loads modules;
** Sets an environment variable;
** Runs the <code>openmp_example</code> application.
* To use hyperthreading, just change <code>--cpus-per-task=40</code> to <code>--cpus-per-task=80</code>.

== Monitoring queued jobs ==

Once the job is incorporated into the queue, there are some commands you can use to monitor its progress.

<ul>
<li><code>squeue</code> or <code>sqc</code> (a caching version of squeue) to show the job queue (<code>squeue -u $USER</code> for just your jobs);</li>
<li><code>squeue -j JOBID</code> to get information on a specific job
(alternatively, <code>scontrol show job JOBID</code>, which is more verbose).</li>
<li><code>squeue --start -j JOBID</code> to get an estimate for when a job will run; these tend not to be very accurate predictions.</li>
<li><code>scancel -i JOBID</code> to cancel the job.</li>
<li><code>jobperf JOBID</code> to get an instantaneous view of the cpu and memory usage of the nodes of the job while it is running.</li>
<li><code>sacct</code> to get information on your recent jobs.</li>
</ul>

Further instructions for monitoring your jobs can be found on the [[Slurm#Monitoring_jobs | Slurm page]]. The [https://my.scinet.utoronto.ca my.SciNet] site is also a very useful tool for monitoring your current and past usage.

= Visualization =
Information about how to use visualization tools on Niagara is available on [[Visualization]] page.

= Support =

* [mailto:support@scinet.utoronto.ca support@scinet.utoronto.ca]
* [mailto:niagara@computecanada.ca niagara@computecanada.ca]

Niagara Quickstart

2022-02-01T19:17:49Z

Gauriaur: /* Logging in */

{{Infobox Computer
|image=[[Image:Niagara.jpg|center|300px|thumb]]
|name=Niagara
|installed=Jan 2018/March 2020
|operatingsystem= CentOS 7.6
|loginnode= niagara.scinet.utoronto.ca
|nnodes= 2,024 nodes (80,960 cores)
|rampernode=188 GiB / 202 GB
|corespernode=40 (80 hyperthreads)
|interconnect=Mellanox Dragonfly+
|vendorcompilers= icc (C) ifort (fortran) icpc (C++)
|queuetype=Slurm
}}

=Specifications=

The Niagara cluster is a large cluster of 2,024 Lenovo SD530 servers each with 40 Intel "Skylake" at 2.4 GHz or 40 Intel "CascadeLake" cores at 2.5 GHz.
The peak performance of the cluster is about 3.6 PFlops (6.25 PFlops theoretical). It was the 53rd fastest supercomputer on the [https://www.top500.org/list/2018/06/?page=1 TOP500 list of June 2018], and is at number 113 on the [https://www.top500.org/lists/top500/list/2021/06/ current list (June 2021)].

Each node of the cluster has 188 GiB / 202 GB RAM per node (at least 4 GiB/core for user jobs). Being designed for large parallel workloads, it has a fast interconnect consisting of EDR InfiniBand in a Dragonfly+ topology with Adaptive Routing. The compute nodes are accessed through a queueing system that allows jobs with a minimum of 15 minutes and a maximum of 24 hours and favours large jobs.

* See the [https://www.youtube.com/watch?v=l-E2CFGh0BE&feature=youtu.be "Intro to Niagara"] recording

More detailed hardware characteristics of the Niagara supercomputer can be found [https://docs.computecanada.ca/wiki/Niagara on this page].

Note: Documentation about the "GPU expansion to Niagara" called "Mist" can be found on [[Mist | its own page]].

= Getting started on Niagara =

Access to Niagara is not enabled automatically for everyone with a Compute Canada account, but anyone with an active Compute Canada account can get their access enabled.

If you have an active Compute Canada account but you do not have access to Niagara yet (e.g. because you are new to SciNet or belong to a group whose primary PI does not have an allocation as granted in the annual [https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions Compute Canada RAC]), go to the [https://ccdb.computecanada.ca/services/opt_in opt-in page on the CCDB site]. After clicking the "Join" button, it usually takes only one or two business days for access to be granted.

Please read this document carefully. The [https://docs.scinet.utoronto.ca/index.php/FAQ FAQ] is also a useful resource. If at any time you require assistance, or if something is unclear, please do not hesitate to [mailto:support@scinet.utoronto.ca contact us].

== Logging in ==

Niagara runs CentOS 7, which is a type of Linux. You will need to be familiar with Linux systems to work on Niagara. If you are not it will be worth your time to review our [https://support.scinet.utoronto.ca/education/browse.php?category=-1&search=scmp101&include=all&filter=Filter Introduction to Linux Shell] class.

As with all SciNet and CC (Compute Canada) compute systems, access to Niagara is done via [[SSH]] (secure shell) only. Open a terminal window (e.g. Connecting with [https://docs.computecanada.ca/wiki/Connecting_with_PuTTY PuTTY] on Windows or Connecting with [https://docs.computecanada.ca/wiki/Connecting_with_MobaXTerm MobaXTerm]), then SSH into the Niagara login nodes with your CC credentials:

$ ssh -Y MYCCUSERNAME@niagara.scinet.utoronto.ca

or

$ ssh -Y MYCCUSERNAME@niagara.computecanada.ca

The first time you login to Niagara, please make sure you are actually accessing Niagara by checking if the login node ssh host key fingerprint matches [[SSH_Changes_in_May_2019 | (See here how)]]. This check prevents you from falling victim of [https://en.wikipedia.org/wiki/Man-in-the-middle_attack man-in-the-middle attacks.]

'''Note:''' As of January 22 2022, authentication is only allowed via SSH keys. [https://docs.computecanada.ca/wiki/SSH_Keys Please refer to this page] to generate your SSH key pair and make sure you use them securely.

* The Niagara login nodes are where you develop, edit, compile, prepare and submit jobs.
* These login nodes are not part of the Niagara compute cluster, but have the same architecture, operating system, and software stack.
* The optional <code>-Y</code> is needed to open windows from the Niagara command-line onto your local X server.
* You can only connect 4 times in a 2-minute window to the login nodes.
* To run on Niagara's compute nodes, you must [[#Submitting_jobs | submit a batch job]].

If you cannot log in, be sure to first check the [https://docs.scinet.utoronto.ca System Status] on this site's front page.

== Your various directories ==

By virtue of your access to Niagara you are granted storage space on the system. There are several directories available to you, each indicated by an associated environment variable.

=== home and scratch ===

You have a home and scratch directory on the system, the paths to which are stored in the environment variables $HOME and $SCRATCH. The locations are of the form

$HOME=/home/g/groupname/myccusername
$SCRATCH=/scratch/g/groupname/myccusername

where groupname is the name of your PI's group, and myccusername is your CC username. For example:

nia-login07:~$ pwd
/home/s/scinet/rzon
nia-login07:~$ cd $SCRATCH
nia-login07:rzon$ pwd
/scratch/s/scinet/rzon

NOTE: home is read-only on compute nodes.

=== project and archive/nearline ===

Users from groups with [https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions RAC storage allocation] will also have a project directory and possible an archive (a.k.a. "nearline") directory, the paths to which are stored in the environment variables $PROJECT and $ARCHIVE. They follow the naming convention:

$PROJECT=/project/g/groupname/myccusername
$ARCHIVE=/archive/g/groupname/myccusername

NOTE: Currently archive space is available only via [[HPSS]], and is not accessible on the Niagara login, compute, or datamover nodes.

'''''IMPORTANT: Future-proof your scripts'''''

When writing your scripts, use the environment variables (<tt>$HOME</tt>, <tt>$SCRATCH</tt>, <tt>$PROJECT</tt>, <tt>$ARCHIVE</tt>) instead of the actual paths! The paths may change in the future.

=== Storage and quotas ===

You should familiarize yourself with the [[Data_Management#Purpose_of_each_file_system | various file systems]], what purpose they serve, and how to properly use them. This table summarizes the various file systems. See the [[Data_Management | Data Management]] page for more details.

{| class="wikitable"
! location
!colspan="2"| quota
!align="right"| block size
! expiration time
! backed up
! on login nodes
! on compute nodes
|-
| $HOME
|colspan="2"| 100 GB / 250,000 files per user
|align="right"| 1 MB
|
| yes
| yes
| read-only
|-
|rowspan="2"| $SCRATCH
|colspan="2"| 25 TB / 6,000,000 file per user
|align="right" rowspan="2" | 16 MB
|rowspan="2"| 2 months
|rowspan="2"| no
|rowspan="2"| yes
|rowspan="2"| yes
|-
|align="right"|50-500TB per group
|align="right"|[[Data_Management#Quotas_and_purging | depending on group size]]
|-
| $PROJECT
|colspan="2"| by group allocation
|align="right"| 16 MB
|
| yes
| yes
| yes
|-
| $ARCHIVE
|colspan="2"| by group (nearline) allocation
|align="right"|
|
| dual-copy
| no
| no
|-
| $BBUFFER
|colspan="2"| 10 TB per user
|align="right"| 1 MB
| very short
| no
| yes
| yes
|}

=== Moving data to Niagara ===

If you need to move data to Niagara for analysis, or when you need to move data off of Niagara, use the following guidelines:
* If your data is less than 10GB, move the data using the login nodes.
* If your data is greater than 10GB, move the data using the datamover nodes nia-datamover1.scinet.utoronto.ca and nia-datamover2.scinet.utoronto.ca .

Details of how to use the datamover nodes can be found on the [[Data_Management#Moving_data | Data Management ]] page.

= Loading software modules =

You have two options for running code on Niagara: use existing software, or [[Niagara_Quickstart#Compiling_on_Niagara:_Example | compile your own]]. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]].

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.
== Software stacks: NiaEnv and CCEnv ==

On Niagara, there are two available software stacks:

=== NiaEnv ===

A [https://docs.scinet.utoronto.ca/index.php/Modules_specific_to_Niagara Niagara software stack] tuned and compiled for this machine. This stack is available by default, but if not, can be reloaded with
<pre>module load NiaEnv</pre>
This loads the default (set of modules), which is currently the 2019b epoch. Before September 1, the default was NiaEnv/2018a. Users are encourage to use the 2019b stack, but to make sure old job scripts or older software installations in your home directory continue to work, you may need to use
<pre>module load NiaEnv/2018a</pre>
You can override the system default for the epoch version by creating a file called <tt>.modulerc</tt> in your home directory with the line <tt>module-version NiaEnv/VERSION default</tt>, e.g. like so:
<pre>
echo "module-version NiaEnv/2019b default" > $HOME/.modulerc
</pre>
After this, subsequent logins and jobs will use the 2019b stack even when the system default is different.
Similarly, you can make an older epoch your personal default, like so
<pre>
echo "module-version NiaEnv/2018a default" > $HOME/.modulerc
</pre>

No modules are loaded by default on Niagara except NiaEnv.

=== CCEnv ===

The same [https://docs.computecanada.ca/wiki/Modules software stack available on Compute Canada's General Purpose clusters] [https://docs.computecanada.ca/wiki/Graham Graham] and [https://docs.computecanada.ca/wiki/Cedar Cedar] can be used on Niagara too, with:
<pre>module load CCEnv</pre>
Or, if you want the same default modules loaded as on Béluga, then do
<pre>module load CCEnv StdEnv</pre>
or, if you want the same default modules loaded as on Cedar and Graham, do
<pre>module load CCEnv arch/avx2 StdEnv</pre>

== Tips for loading software ==

* We advise '''''against''''' loading modules in your .bashrc. This can lead to very confusing behaviour under certain circumstances. Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>intel</code> → <code>intel/2018.2</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Modules often require other modules to be loaded first. Solve these dependencies by using [[Using_modules#Module_spider | <code>module spider</code>]].

= Available compilers and interpreters =

* For most compiled software, one should use the Intel compilers (<tt>icc</tt> for C, <tt>icpc</tt> for C++, and <tt>ifort</tt> for Fortran). Loading an <tt>intel</tt> module makes these available.
* The GNU compiler suite (<tt>gcc, g++, gfortran</tt>) is also available, if you load one of the <tt>gcc</tt> modules.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> or <tt>intelmpi</tt> module.
* Open source interpreted, interactive software is also available:
** [[Python]]
** [[R]]
** Julia
** [[Octave]]

Please visit the corresponding page for details on using these tools. For information on running MATLAB applications on Niagara, visit [[MATLAB| this page]].

= Using Commercial Software =

May I use commercial software on Niagara?
* Possibly, but you have to bring your own license for it. You can connect to an external license server using [[SSH_Tunneling | ssh tunneling]].
* SciNet and Compute Canada have an extremely large and broad user base of thousands of users, so we cannot provide licenses for everyone's favorite software.
* Thus, the only freely available commercial software installed on Niagara is software that can benefit everyone: Compilers, math libraries and debuggers.
* That means no [[MATLAB]], Gaussian, IDL,
* Open source alternatives like Octave, [[Python]], and [[R]] are available.
* We are happy to help you to install commercial software for which you have a license.
* In some cases, if you have a license, you can use software in the Compute Canada stack.
The list of commercial software which is installed on Niagara, for which you will need a license to use, can be found on the [[Commercial_software | commercial software page]].

= Compiling on Niagara: Example =

Suppose one wants to compile an application from two c source files, appl.c and module.c, which use the Math Kernel Library. This is an example of how this would be done:
<source lang="bash">
nia-login07:~$ module load NiaEnv/2019b
nia-login07:~$ module list
Currently Loaded Modules:
1) NiaEnv/2019b (S)
Where:
S: Module is Sticky, requires --force to unload or purge

nia-login07:~$ module load intel/2019u4

nia-login07:~$ ls
appl.c module.c

nia-login07:~$ icc -c -O3 -xHost -o appl.o appl.c
nia-login07:~$ icc -c -O3 -xHost -o module.o module.c
nia-login07:~$ icc -o appl module.o appl.o -mkl

nia-login07:~$ ./appl
</source>
Note:
* The optimization flags -O3 -xHost allow the Intel compiler to use instructions specific to the architecture CPU that is present (instead of for more generic x86_64 CPUs).
* Linking with the Intel Math Kernel Library (MKL) is easy when using the intel compiler, it just requires the -mkl flags.
* If compiling with gcc, the optimization flags would be -O3 -march=native. For the way to link with the MKL, it is suggested to use the [https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor MKL link line advisor].

= Testing and Debugging =

You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.
* Small test jobs can be run on the login nodes. Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than a couple of cores.
* You can run the [[Parallel Debugging with DDT|DDT]] debugger on the login nodes after <code>module load ddt</code>.
* Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
nia-login07:~$ debugjob --clean N
where N is the number of nodes, If N=1, this gives an interactive session one 1 hour, when N=4 (the maximum), it gives you 22 minutes. The <tt>--clean</tt> argument is optional but recommended as it will start the session without any modules loaded, thus mimicking more closely what happens when you submit a job script.

Finally, if your debugjob process takes more than 1 hour, you can request an interactive job from the regular queue using the salloc command. Note, however, that this may take some time to run, since it will be part of the regular queue, and will be run when the scheduler decides.
nia-login07:~$ salloc --nodes N --time=M:00:00 --x11
where N is again the number of nodes, and M is the number of hours you wish the job to run.
The <tt>--x11</tt> is required if you need to use graphics while testing your code through salloc, e.g. when using a debugger such as [[Parallel Debugging with DDT|DDT]] or DDD, See the [[Testing_With_Graphics | Testing with graphics]] page for the options in that case.

= Submitting jobs =




Once you have compiled and tested your code or workflow on the Niagara login nodes, and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. Your jobs will run on some of Niagara's 1548 compute nodes. When and where your job runs is determined by the scheduler.

Niagara uses SLURM as its job scheduler. More-advanced details of how to interact with the scheduler can be found on the [[Slurm | Slurm page]].

You submit jobs from a login node by passing a script to the sbatch command:

nia-login07:scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course. Note that you must submit your job from a login node. You cannot submit jobs from the datamover nodes.

In most cases, you should not submit from your $HOME directory, but rather, from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

Jobs will run under your group's RRG allocation, or, if the your group has none, under a RAS allocation (previously called `default' allocation).

Some example job scripts can be found below.

Keep in mind:
* Scheduling is by node, so in multiples of 40 cores.
* Your job's maximum walltime is 24 hours.
* Jobs must write their output to your scratch or project directory (home is read-only on compute nodes).
* Compute nodes have no internet access.
* Your job script will not remember the modules you have loaded, so it needs to contain "module load" commands of all the required modules (see examples below).
* [[Data_Management#Moving_data | Move your data]] to Niagara before you submit your job.

== Scheduling by Node ==

On many systems that use SLURM, the scheduler will deduce from the specifications of the number of tasks and the number of cpus-per-node what resources should be allocated. On Niagara things are a bit different.
* All job resource requests on Niagara are scheduled as a multiple of '''nodes'''.
* The nodes that your jobs run on are exclusively yours, for as long as the job is running on them.
** No other users are running anything on them.
** You can [[SSH]] into them to see how things are going.
* Whatever your requests to the scheduler, it will always be translated into a multiple of nodes allocated to your job.
* Memory requests to the scheduler are of no use. Your job always gets N x 202GB of RAM, where N is the number of nodes and 202GB is the amount of memory on the node.
* If you run serial jobs you must still use all 40 cores on the node. Visit the [[Running_Serial_Jobs_on_Niagara | serial jobs]] page for examples of how to do this.
* Since there are 40 cores per node, your job should use N x 40 cores. If you do not, we will contact you to help you optimize your workflow. Or you can [mailto:support@scinet.utoronto.ca contact us] to get assistance.

== Limits ==

There are limits to the size and duration of your jobs, the number of jobs you can run and the number of jobs you can have queued. It matters whether a user is part of a group with a [https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions/ Resources for Research Group allocation] or not. It also matters in which 'partition' the job runs. 'Partitions' are SLURM-speak for use cases. You specify the partition with the <tt>-p</tt> parameter to <tt>sbatch</tt> or <tt>salloc</tt>, but if you do not specify one, your job will run in the <tt>compute</tt> partition, which is the most common case.

{| class="wikitable"
!Usage
!Partition
!Limit on Running jobs
!Limit on Submitted jobs (incl. running)
!Min. size of jobs
!Max. size of jobs
!Min. walltime
!Max. walltime
|-
|Compute jobs ||compute || 50 || 1000 || 1 node (40 cores) || default: 20 nodes (800 cores) with allocation: 1000 nodes (40000 cores)|| 15 minutes || 24 hours
|-
|Testing or troubleshooting || debug || 1 || 1 || 1 node (40 cores) || 4 nodes (160 cores)|| N/A || 1 hour
|-
|Archiving or retrieving data in [[HPSS]]|| archivelong || 2 per user (5 in total) || 10 per user || N/A || N/A|| 15 minutes || 72 hours
|-
|Inspecting archived data, small archival actions in [[HPSS]] || archiveshort vfsshort || 2 per user|| 10 per user || N/A || N/A || 15 minutes || 1 hour
|}

Even if you respect these limits, your jobs will still have to wait in the queue. The waiting time depends on many factors such as your group's allocation amount, how much allocation has been used in the recent past, the number of requested nodes and walltime, and how many other jobs are waiting in the queue.

== File Input/Output Tips ==

It is important to understand the file systems, so as to perform your file I/O (Input/Output) responsibly. Refer to the [[Data_Management | Data Management]] page for details about the file systems.
* Your files can be seen on all Niagara login and compute nodes.
* $HOME, $SCRATCH, and $PROJECT all use the parallel file system called GPFS.
* GPFS is a high-performance file system which provides rapid reads and writes to large data sets in parallel from many nodes.
* Accessing data sets which consist of many, small files leads to poor performance on GPFS.
* Avoid reading and writing lots of small amounts of data to disk. Many small files on the system waste space and are slower to access, read and write. If you must write many small files, use [[User_Ramdisk | ramdisk]].
* Write data out in a binary format. This is faster and takes less space.
* The [[Burst Buffer]] is another option for I/O heavy-jobs and for speeding up [[Checkpoints|checkpoints]].

== Example submission script (MPI) ==

<source lang="bash">#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=1:00:00
#SBATCH --job-name=mpi_job
#SBATCH --output=mpi_output_%j.txt
#SBATCH --mail-type=FAIL

cd $SLURM_SUBMIT_DIR

module load NiaEnv/2019b
module load intel/2019u4
module load openmpi/4.0.1

mpirun ./mpi_example
# or "srun ./mpi_example"
</source>
Submit this script from your scratch directory with the command:

nia-login07:scratch$ sbatch mpi_job.sh

<ul>
<li>First line indicates that this is a bash script.</li>
<li>Lines starting with <code>#SBATCH</code> go to SLURM.</li>
<li>sbatch reads these lines as a job request (which it gives the name <code>mpi_job</code>)</li>
<li>In this case, SLURM looks for 2 nodes each running 40 tasks (for a total of 80 tasks), for 1 hour</li>
<li>Note that the mpifun flag "--ppn" (processors per node) is ignored.</li>
<li>Once it found such a node, it runs the script:
<ul>
<li>Change to the submission directory;</li>
<li>Loads modules;</li>
<li>Runs the <code>mpi_example</code> application (SLURM will inform mpirun or srun on how many processes to run).
</li>
</ul>
<li>To use hyperthreading, just change <code>--ntasks-per-node=40</code> to <code>--ntasks-per-node=80</code>, and add <code>--bind-to none</code> to the mpirun command (the latter is necessary for OpenMPI only, not when using IntelMPI).</li>
</ul>

== Example submission script (OpenMP) ==

<source lang="bash">#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=40
#SBATCH --time=1:00:00
#SBATCH --job-name=openmp_job
#SBATCH --output=openmp_output_%j.txt
#SBATCH --mail-type=FAIL

cd $SLURM_SUBMIT_DIR

module load NiaEnv/2019b
module load intel/2019u4

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

./openmp_example
# or "srun ./openmp_example".
</source>
Submit this script from your scratch directory with the command:

nia-login07:~$ sbatch openmp_job.sh

* First line indicates that this is a bash script.
* Lines starting with <code>#SBATCH</code> go to SLURM.
* sbatch reads these lines as a job request (which it gives the name <code>openmp_job</code>) .
* In this case, SLURM looks for one node with 40 cores to be run inside one task, for 1 hour.
* Once it found such a node, it runs the script:
** Change to the submission directory;
** Loads modules;
** Sets an environment variable;
** Runs the <code>openmp_example</code> application.
* To use hyperthreading, just change <code>--cpus-per-task=40</code> to <code>--cpus-per-task=80</code>.

== Monitoring queued jobs ==

Once the job is incorporated into the queue, there are some commands you can use to monitor its progress.

<ul>
<li><code>squeue</code> or <code>sqc</code> (a caching version of squeue) to show the job queue (<code>squeue -u $USER</code> for just your jobs);</li>
<li><code>squeue -j JOBID</code> to get information on a specific job
(alternatively, <code>scontrol show job JOBID</code>, which is more verbose).</li>
<li><code>squeue --start -j JOBID</code> to get an estimate for when a job will run; these tend not to be very accurate predictions.</li>
<li><code>scancel -i JOBID</code> to cancel the job.</li>
<li><code>jobperf JOBID</code> to get an instantaneous view of the cpu and memory usage of the nodes of the job while it is running.</li>
<li><code>sacct</code> to get information on your recent jobs.</li>
</ul>

Further instructions for monitoring your jobs can be found on the [[Slurm#Monitoring_jobs | Slurm page]]. The [https://my.scinet.utoronto.ca my.SciNet] site is also a very useful tool for monitoring your current and past usage.

= Visualization =
Information about how to use visualization tools on Niagara is available on [[Visualization]] page.

= Support =

* [mailto:support@scinet.utoronto.ca support@scinet.utoronto.ca]
* [mailto:niagara@computecanada.ca niagara@computecanada.ca]

Main Page

2021-12-08T14:31:57Z

Gauriaur:

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up |Niagara|Niagara_Quickstart}}
|{{Up |Mist|Mist}}
|{{Up |Teach|Teach}}
|{{Up |Rouge|Rouge}}
|-
|{{Up |Jupyter Hub|Jupyter_Hub}}
|{{Up |Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up |File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up |Burst Buffer|Burst_Buffer}}
|-
|{{Up|HPSS|HPSS}}
|{{Up |Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up |External Network|Niagara_Quickstart#Logging_in}}
|{{Up |Globus |Globus}}
|}



Wed Dec 8 2:30 EST PM 2021 Planned maintenance. Datacenter external connections will be interrupted for approximately 30 minutes. Jobs in the queue will continue to run, and access should return by 3:00 PM EST.

Mon Dec 6 9:28 EST AM 2021A power glitch caused many compute nodes to reboot. Please check your jobs, and in case of failure, resubmit.

Mon Nov 22 1:40 EST PM 2021 The Mist login node is back.

Mon Nov 22 12:40 EST PM 2021 The Mist login node is experiencing issues, we are investigating.

Fri Nov 5 19:35 EDT 2021 The filesystem issue from earlier in the afternoon is resolved.

Fri Nov 5 16:58 EDT 2021 We are experiencing filesystem issues, login to the clusters may not be possible until they are resolved.

Tue Oct 19 noon EDT - Thu Oct 21 noon EDT: Niagara at Scale: Only users of selected projects run at large scale during these 48 hours. Other users can still login and access their files, and submit jobs for after the event. SOSCIP and Mist users are not affected.

Tue Oct 12 14:30 EDT 2021 Mist login node is back up.

Tue Oct 12 12:30 EDT 2021 Mist login node is down for maintenance.

Mon Sep 27 16:11 EDT 2021 HPSS is back online.

Wed Sep 23 17:23 EDT 2021 Systems being brought back online. HPSS may be down for some more days.


{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH keys]]
* [[SSH Tunneling]]
* [[SSH#Two-Factor_authentication|Two-Factor Authentication]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2021-12-08T14:31:37Z

Gauriaur:

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up |Niagara|Niagara_Quickstart}}
|{{Up |Mist|Mist}}
|{{Up |Teach|Teach}}
|{{Up |Rouge|Rouge}}
|-
|{{Up |Jupyter Hub|Jupyter_Hub}}
|{{Up |Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up |File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up |Burst Buffer|Burst_Buffer}}
|-
|{{Up|HPSS|HPSS}}
|{{Up |Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up |External Network|Niagara_Quickstart#Logging_in}}
|{{Up |Globus |Globus}}
|}



Wed 8 2:30 EST PM 2021 Planned maintenance. Datacenter external connections will be interrupted for approximately 30 minutes. Jobs in the queue will continue to run, and access should return by 3:00 PM EST.

Mon Dec 6 9:28 EST AM 2021A power glitch caused many compute nodes to reboot. Please check your jobs, and in case of failure, resubmit.

Mon Nov 22 1:40 EST PM 2021 The Mist login node is back.

Mon Nov 22 12:40 EST PM 2021 The Mist login node is experiencing issues, we are investigating.

Fri Nov 5 19:35 EDT 2021 The filesystem issue from earlier in the afternoon is resolved.

Fri Nov 5 16:58 EDT 2021 We are experiencing filesystem issues, login to the clusters may not be possible until they are resolved.

Tue Oct 19 noon EDT - Thu Oct 21 noon EDT: Niagara at Scale: Only users of selected projects run at large scale during these 48 hours. Other users can still login and access their files, and submit jobs for after the event. SOSCIP and Mist users are not affected.

Tue Oct 12 14:30 EDT 2021 Mist login node is back up.

Tue Oct 12 12:30 EDT 2021 Mist login node is down for maintenance.

Mon Sep 27 16:11 EDT 2021 HPSS is back online.

Wed Sep 23 17:23 EDT 2021 Systems being brought back online. HPSS may be down for some more days.


{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH keys]]
* [[SSH Tunneling]]
* [[SSH#Two-Factor_authentication|Two-Factor Authentication]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

SSH Changes in May 2019

2021-06-13T19:21:36Z

Gauriaur:

=SSH Changes in May 2019 (English)=

(La version francaise suit.)

== What Changed? ==

During the 29-30 May 2019 shutdown, we made the following ssh security improvements on Niagara:

# Disabled certain weak encryption algorithms.
# Disabled certain weak public key types.
# Regenerated Niagara's host keys.

== Updating your client's known host list ==

The first time you login to Niagara after the shutdown, you will probably see the following warning message:

<pre>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:SauX2nL+Yso9KBo2Ca6GH/V9cSFLFXwxOECGWXZ5pxc.
Please contact your system administrator.
Add correct host key in /home/username/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/username/.ssh/known_hosts:109
ED25519 host key for niagara.scinet.utoronto.ca has changed and you have requested strict checking.
Host key verification failed.
Killed by signal 1.
</pre>

This warning is displayed because the host keys on Niagara changed to increase the data centre's security, and ssh clients remember old host keys to prevent [https://en.wikipedia.org/wiki/Man-in-the-middle_attack "man-in-the-middle" attacks].

You may also get a warning regarding "DNS spoofing", which is related to the same change.

If you are using MobaXTerm, Putty, or WinSCP as your ssh (or scp) client under Windows, the warning will appear in a pop-up window and will allow you to accept the new host key by clicking "Yes".

If you are using the command line ssh command on macOS, Linux, GitBash or Cygwin, you should tell your system to "forget" the old host keys, by running the following commands:

ssh-keygen -R niagara.scinet.utoronto.ca
ssh-keygen -R niagara.computecanada.ca
ssh-keygen -R 142.150.188.70
ssh-keygen -R nia-datamover1.scinet.utoronto.ca
ssh-keygen -R nia-datamover2.scinet.utoronto.ca
ssh-keygen -R 142.150.188.131
ssh-keygen -R 142.150.188.132

Afterwards, the next time you ssh to Niagara you'll be asked to confirm the new host keys, e.g.:

$ ssh niagara.scinet.utoronto.ca
The authenticity of host 'niagara.scinet.utoronto.ca (142.150.188.70)' can't be established.
ED25519 key fingerprint is SHA256:SauX2nL+Yso9KBo2Ca6GH/V9cSFLFXwxOECGWXZ5pxc.
ED25519 key fingerprint is MD5:b4:ae:76:a5:2b:37:8d:57:06:0e:9a:de:62:00:26:be.
Are you sure you want to continue connecting (yes/no)?

Make sure the fingerprints are correct! You'll either see the above ED25519 fingerprints, or the following RSA fingerprints:

RSA key fingerprint is SHA256:k6YEhYsI73M+NJIpZ8yF+wqWeuXS9avNs2s5QS/0VhU.
RSA key fingerprint is MD5:98:e7:7a:07:89:ef:3f:d8:68:3d:47:9c:6e:a6:71:5e.

If the fingerprints don't match, someone may be trying to hijack your connection.

== Troubleshooting ==

=== I can't connect! ===

If you see one of the following error messages:

Unable to negotiate with 142.150.188.70 port 22: no matching cipher found.
Unable to negotiate with 142.150.188.70 port 22: no matching key exchange method found.
Unable to negotiate with 142.150.188.70 port 22: no matching mac found.

you need to upgrade your ssh client.

=== My SSH key no longer works ===

If you're being asked for a password, but were using SSH keys,
it's because 1024-bit DSA & RSA keys have been disabled.

You need to generate a new stronger key, see the [[SSH keys]] page for details.

(What follows is the French version of the above information).

=Les changement de SSH à mai 2019 (version française)=

==Qu'est-ce qui a changé?==

Pendant l'arrêt planifié des 29 et 30 mai 2019, nous avons mis en place
les mesures suivantes à Niagara pour améliorer sa securité:

# Certaines méthodes de chiffrement faibles ont été désactivées.
# Certains types de clé publique SSH ont été désactivés.
# Les clés d'hôte de Niagara ont été régénérées.

==Mettre à jour la liste de clés d'hôte de votre client SSH==

Lorsque vous vous reconnectez à Niagara pour la première fois après
l'arrêt de maintenance, vous verrez probablement le message suivant:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:SauX2nL+Yso9KBo2Ca6GH/V9cSFLFXwxOECGWXZ5pxc.
Please contact your system administrator.
Add correct host key in /home/username/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/username/.ssh/known_hosts:109
ED25519 host key for niagara.scinet.utoronto.ca has changed and you have requested strict checking.
Host key verification failed.
Killed by signal 1.

Ce message d'avertissement s'affiche parce que les clés d'hôte de Niagara ont été regénérées pour améliorer le securité du centre de données, mais le client SSH a stocké les anciennes clés d'hôte pour empêcher une [https://fr.wikipedia.org/wiki/Attaque_de_l%27homme_du_milieu « attaque de l'homme du milieu »].

C'est possible que vous voyiez aussi un message d'avertissement indiquant « DNS spoofing ».

Si vous utilisez un des outils de connexion MobaXTerm, PuTTY et WinSCP sous Windows, le message d'avertissment s'affiche dans une fenêtre contextuelle qui offre l'option d'accepter la nouvelle clé d'hôte en appuyant sur le bouton approprié.

Si vous utilisez la commande « ssh » dans une fenêtre de terminal sur macOS, Linux, GitBash ou Cygwin, vous devez dire à votre système d'oublier les anciennes clés d'hôte avec les commandes suivantes:

$ ssh-keygen -R niagara.scinet.utoronto.ca
$ ssh-keygen -R niagara.computecanada.ca
$ ssh-keygen -R 142.150.188.70
$ ssh-keygen -R nia-datamover1.scinet.utoronto.ca
$ ssh-keygen -R nia-datamover2.scinet.utoronto.ca
$ ssh-keygen -R 142.150.188.131
$ ssh-keygen -R 142.150.188.132

Ensuite, lors de votre prochaine visite à Niagara, vous devrez confirmer les nouvelles clés d’hôte, par exemple:

$ ssh niagara.scinet.utoronto.ca
The authenticity of host 'niagara.scinet.utoronto.ca (142.150.188.70)' can't be established.
ED25519 key fingerprint is SHA256:SauX2nL+Yso9KBo2Ca6GH/V9cSFLFXwxOECGWXZ5pxc.
ED25519 key fingerprint is MD5:b4:ae:76:a5:2b:37:8d:57:06:0e:9a:de:62:00:26:be.
Are you sure you want to continue connecting (yes/no)?

Assurez-vous que les empreintes digitales (« fingerprints ») sont correctes ! Vous verrez soit les empreintes ED25519 ci-dessus, soit les empreintes RSA suivantes:

RSA key fingerprint is SHA256:k6YEhYsI73M+NJIpZ8yF+wqWeuXS9avNs2s5QS/0VhU.
RSA key fingerprint is MD5:98:e7:7a:07:89:ef:3f:d8:68:3d:47:9c:6e:a6:71:5e.

Si les empreintes digitales ne correspondent pas, il est possible que quelqu'un tente de détourner votre connexion.

==Dépannage==

===Je n'arrive pas à me connecter!===

Si vous voyez l'un des messages d'erreur suivants:

Unable to negotiate with 142.150.188.70 port 22: no matching cipher found.
Unable to negotiate with 142.150.188.70 port 22: no matching key exchange method found.
Unable to negotiate with 142.150.188.70 port 22: no matching mac found.

vous devez mettre à jour le client ssh.

===Ma clé SSH ne fonctionne plus===

Si un mot de passe vous est demandé alors que vous utilisiez auparavant des clés SSH, c'est probablement parce que les clés DSA et RSA 1024 bits ont été désactivées.

Vous devez générer une nouvelle clé plus puissante, voir la page [[SSH keys]] pour plus de détails.