SciNet Users Documentation - User contributions [en]

Main Page

2026-06-10T18:32:27Z

Ymeiron:

2025-12-05T22:36:49Z

Ymeiron: /* System Status */

Main Page

2025-12-05T15:19:30Z

Ymeiron: /* System Status */

Main Page

2025-12-02T21:38:44Z

Ymeiron: Added S4H

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Down | Niagara|Niagara_Quickstart}}
|{{Down | Teach|Teach}}
|{{Down | Rouge|Rouge}}
|-
|{{Partial | OnDemand|Open_OnDemand_Quickstart}}
|{{Up3 | Scheduler|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Submitting_jobs_to_the_scheduler}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Up | HPSS|HPSS}}
|{{Up3 | Login Nodes|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|{{Up | Globus |Globus}}
|-
|{{Up | Balam|Balam}}
|{{Up3 | Cvmfs|https://docs.alliancecan.ca/wiki/Standard_software_environments}}
|{{Up | S4H | S4H}}
|}

'''Sat Nov 29, 2025, 00:40 am:''' There has been a problem with the water chiller. Some systems are offline.

'''Wed Nov 5, 2025, 12:55 pm:''' Balam is back online.

'''Wed Nov 5, 2025, 10:00 am:''' Open OnDemand is back online.

'''Tue Nov 4, 2025, 11:00 pm:''' Most of the work is done, data movers, Globus, and HPSS are back online. Remaining services will be worked on tomorrow.

'''Tue Nov 4, 2025, 8:30 am:''' Scheduled network maintenance. Trillium cluster is *not* affected.

'''Tue Oct 21, 2025, 17:30 am:''' Balam maintenance finished.

'''Tue Oct 21, 2025, 7:00 am:''' Balam maintenance day.

'''Thu Oct 15, 2025, 3:55 pm:''' Trillium inbound connections through trillium.alliancecan.ca or trillium.scinet.utoronto.ca are working again.

'''Thu Oct 15, 2025, 3:05 pm:''' Trillium is experiencing external network issues for both incoming traffic. Please try: ssh USERNAME@tri-login01.scinet.utoronto.ca in the meantime.

'''Thu Oct 06, 2025, 8:00 pm:''' HPSS is fully functional. You may submit archive jobs from trillium login nodes, datamovers and robots.

'''Thu Oct 03, 2025, 6:30 pm:''' HPSS is back online, and already accessible via alliancecan#hpss Globus endpoint. Directory tree now follows the other Alliance clusters. We're still working on job submission via Slurm

'''Thu Oct 01, 2025, 0:00 am:''' Niagara compute nodes are now unavailable for regular users. The login nodes will remain available for a while to allow a few last data transfers, although transfers from the Niagara file systems to Trillium are best done on nia-dm1.scinet.utoronto.ca.

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

S4H

2025-12-02T21:36:40Z

Ymeiron: Page created

= Introduction =
S4H (formerly SciNet4Health) is our secure computing environment pilot, providing users with the ability to run [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium] jobs on confidential data. This subsystem is comprised of a dedicated login node and a storage appliance, but it is highly integrated with Trillium. Security concerns are addressed by

* Hardened access
* Encryption at rest
* Group isolation
* Data egress control (optional)

Usage of S4H is by request only. Access must be requested by a principle investigator (PI) on behalf of their group members (i.e. sponsored users on CCDB).

= Policies =
Each user is assigned one of three policies:

* '''Permissive:''' the user may connect to the login node using SSH from pre-approved source IP addresses, and has unrestricted internet access from the login node
* '''Restrictive:''' the user may connect to the login node using SSH from pre-approved source IP addresses, but internet access from the login node is restricted
* '''Prohibitive:''' the user may only connect to the login node using a remote desktop client program from pre-approved source IP addresses, and internet access from the login node is restricted

If you don't know what policy you belong to, you should ask your PI.

= Login =
== Direct ==
Users with permission to connect directly to the login node (permissive and restrictive policies) should first make sure that they are able to login to Trillium (i.e. they have uploaded an SSH public key to, and set up second factor authentication on CCDB). If access to Trillium is successful, use the same username and SSH key to login to the following address:

<pre>
s4h.scinet.utoronto.ca
</pre>

You should be prompted for the second factor, like in Trillium.

The connection must be made from one of the '''IP addresses pre-approved by the PI''' for that user (e.g. a workstation or a jump host in your lab).

== Through the graphical gateway ==
Users with permission to connect through the graphical gateway should use an RDP-enabled remote desktop client and login to the following address using their CCDB username and password:

<pre>
s4h-ggw.scinet.utoronto.ca
</pre>

It is recommended to set the display resolution to 1600×900 if the resolution is not picked up automatically by the program.

Additionally, a "pre-login" step has to be performed. In this step, an SSH ''agent'' must be forwarded to the above address. The graphical gateway asks the user's SSH agent program to perform the authentication (i.e. the workstation where the user is connecting from has the private key and it has been added to the agent). In a sense this is a 3-factor authentication: one needs the password, the SSH private key, and have either a YubiKey or the Duo mobile app registered with CCDB. Here is an example of this process:

<pre>
eval $(ssh-agent)
ssh-add /home/alice/.ssh/ccdb_ed25519
ssh -T -A alice@s4h-ggw.scinet.utoronto.ca
</pre>

Note that no shell access is expected after the <code>ssh</code> command; but the window on the remote desktop program should now prompt for the YubiKey passcode or Duo mobile app push. Once that is done as well, the SSH client will print _Login successful_ and quit.

The connection must be made from one of the '''IP addresses pre-approved by the PI''' for that user (e.g. a workstation or a jump host in your lab).

= Storage =
== Directories ==
Trillium file systems are accessible via their usual paths but are read-only on S4H (to prevent accidentally saving sensitive data there). Instead, home, scratch, and project spaces are provided on alternative paths under <code>/s4h</code> (indicating the encrypted storage appliance). If the user "alice" belongs to the group "def-bob" on S4H, their home directory (which can be expanded from <code>~</code> or <code>$HOME</code>) will be located in <code>/s4h/def-bob/home/alice</code> and similarly their scratch directory (can be expanded from <code>$SCRATCH</code>) will be in <code>/s4h/def-bob/scratch/alice</code>. The project directory is <code>/s4h/def-bob/project</code>, users may create their own directories there as needed.

The environment variables <code>$TRIHOME</code>, <code>$TRISCRATCH</code>, and <code>$TRIPROJECT</code> expand to the corresponding file system paths for Trillium (as noted above, they are read-only on S4H)

== Data transfer ==
For users in the permissive and restrictive policies, please use an SSH-based program (such as <code>scp</code> or <code>rsync</code>) to transfer data directly in and out of the S4H login node. There is no dedicated datamover for S4H.

Under the prohibitive policy, users may not transfer sensitive data in and out of S4H. They may only upload non-sensitive data to Trillium (storage not encrypted at rest) where they can be accessed from S4H. For egress purposes, the PI should designate at least one user (could be themselves) that is not under the prohibitive policy. Other users in the group could share files for egress with the designated user (e.g. by putting them in the group's project directory).

== Data policies ==
It is important to understand that:
* Within a group, file access is managed by traditional POSIX permissions and access-control lists, like on Trillium. In case users in a group are working on separate sub-projects where there should not be mutual access, it is their responsibility to make sure that permissions are set up correctly.
* There is no way to facilitate cross-group file sharing of sensitive files on S4H; each group has a different encryption key and the system is set up so that a compute node can only use one key at a time.
* No backup is provided for encrypted storage; deletion is irreversible. This ensure that data are securely disposed of in compliance with a provision found in many data sharing agreements.

= Software =
Same as Trillium.

Note that you may use software (including, for example, Python virtual environments) that you installed in your Trillium file systems on S4H (but not vice versa). This could be useful for users under the restrictive or prohibitive policy, that may otherwise have difficulty installing software in their encrypted storage spaces.

= Submitting jobs =
This is largely the same as Trillium. Note however that job metadata are not kept confidential! In particular, the submitting user, work directory, job name, comment, and command should be ''considered public information'' and the users must not include any sensitive information in these.

Trillium Quickstart

2025-08-07T21:28:41Z

Ymeiron: Added another note about the address of the GPU system

{{Infobox Computer
|image=[[Image:Trillium.jpg|center|300px|thumb]]
|name=Trillium
|installed=Aug 2025
|operatingsystem= Rocky Linux 9.6
|loginnode= trillium.scinet.utoronto.ca, trillium-gpu.scinet.utoronto.ca
|nnodes= 1284 nodes (240768 cores)
|rampernode= 768 GB
|corespernode= 192 (CPU nodes) and 96 (GPU nodes)
|interconnect=Mellanox Dragonfly+
|queuetype=Slurm
}}

__TOC__

= System Overview =

The Trillium system is a state-of-the-art high performance computing platform, consisting of three main components:

1. CPU Subcluster
* ~240,000 cores across homogeneous CPU nodes
* Non-blocking 400 Gb/s NDR InfiniBand interconnect
* Ideal for large-scale parallel workloads

2. GPU Subcluster
* 61 GPU nodes, each with 4 x NVIDIA H100 (SXM) GPUs
* 800 Gb/s bandwidth per node (200 Gb/s per GPU) over InfiniBand
* Optimized for AI/ML and accelerated science workloads
* Note: This subcluster is in high demand and not ideal for training extremely large models (multi-100B parameters)
* To access, SSH into <code>trillium-gpu.scinet.utoronto.ca</code> from outside, or to <code>trig-login01</code> from other Trillium nodes.

3. Storage System
* Unified 29 PB VAST NVMe storage for all workloads
* No tiering — all flash-based for consistent performance
* Accessible via POSIX or S3 under a unified namespace

== Specifications ==

The Trillium cluster is a large cluster comprised of two types of nodes:

{| class="wikitable sortable"
! nodes !! cores !! available memory !! CPU !! GPU
|-
| 1224 || 192 || 768GB DDR5 ||2 x AMD EPYC 9655 (Zen 5) @ 2.6 GHz, 384MB cache L3 ||
|-
| 60 || 96 || 768GB DDR5 || 1 x AMD EPYC 9654 (Zen 4) @ 2.4 GHz, 384MB cache L3 || 4 x NVIDIA H100 SXM (80 GB memory)
|}

Each node of the cluster has 768 GB RAM per node. Being designed for large parallel workloads, it has a fast interconnect consisting of NDR InfiniBand in a Dragonfly+ topology with Adaptive Routing. The compute nodes are accessed through a queueing system that allows jobs with a minimum of 15 minutes and a maximum of 24 hours.

== Storage System ==

Trillium features a unified high-performance storage system based on the VAST platform, with no tiering. It serves the following directories:

* <code>/home</code> – For personal files and configurations.
* <code>/scratch</code> – High-speed, temporary storage for job data.
* <code>/project</code> – Shared storage for project teams and collaborations.

The storage is accessible via the NDR InfiniBand fabric for maximum performance across all workloads.

= Getting started on Trillium =

Access to Trillium is not enabled automatically for everyone with an account with the {{DigitalResearchAllianceOfCanada}}, but anyone with an active Alliance account can get their access enabled.

Trillium is not automatically available to all Alliance account holders. If you are new to SciNet or your Supervisor/PI does not hold a current {{Alliance}} [https://alliancecan.ca/en/services/advanced-research-computing/research-portal/accessing-resources/resource-allocation-competitions RAC] allocation, you will need to request access on the [https://ccdb.alliancecan.ca/me/access_systems Access Systems] page on the CCDB site. After clicking the "I request access" button, it usually takes only one or two business days for access to be granted.

You can check if you already have Trillium access by attempting to log in. If you receive a "Permission denied" error (and your SSH key is correctly set up), you may need to opt in.

Please read this document carefully. The [https://docs.scinet.utoronto.ca/index.php/FAQ FAQ] is also a useful resource. If at any time you require assistance, or if something is unclear, please do not hesitate to [mailto:support@scinet.utoronto.ca contact us].

== Logging in ==

Trillium runs Rocky Linux 9.6, which is a type of Linux. You will need to be familiar with Linux systems to work on Trillium. If you are not it will be worth your time to review our [https://support.scinet.utoronto.ca/education/browse.php?category=-1&search=scmp101&include=all&filter=Filter Introduction to Linux Shell] class.

As with all SciNet and {{Alliance}} compute systems, access to Trillium is done via [[SSH]] (secure shell) only and authentication is only allowed via SSH keys. [https://docs.alliancecan.ca/wiki/SSH_Keys Please refer to this page] to generate your SSH key pair and make sure you use them securely.

Open a terminal window (e.g. Connecting with [https://docs.alliancecan.ca/wiki/Connecting_with_PuTTY PuTTY] on Windows or Connecting with [https://docs.alliancecan.ca/wiki/Connecting_with_MobaXTerm MobaXTerm]), then SSH into the Trillium login nodes with your {{Alliance}} credentials:

$ ssh -i /path/to/ssh_private_key -Y MYALLIANCEUSERNAME@trillium.scinet.utoronto.ca

* The Trillium login nodes are where you develop, edit, compile, prepare and submit jobs.
* These login nodes are not part of the Trillium compute cluster, but have the same architecture, operating system, and software stack.
* The optional <code>-Y</code> enables X11 forwarding, allowing graphical programs to open windows on your local computer.
* To run on Trillium compute nodes, you must [[#Submitting_jobs | submit a batch job]].

If you cannot log in, be sure to first check the [https://docs.scinet.utoronto.ca System Status] on this site's front page.

Note: We plan to add browser access to Trillium via Open OnDemand in the future. In the meantime you can still access our existing Open OnDemand deployment by following the instructions in our [https://docs.scinet.utoronto.ca/index.php/Open_OnDemand_Quickstart quickstart guide].

== Software Environment ==

Trillium uses the '''environment modules''' system to manage compilers, libraries, and other software packages. Modules dynamically modify your environment (e.g., <code>PATH</code>, <code>LD_LIBRARY_PATH</code>) so you can access different versions of software without conflicts.

A detailed explanation can be [[Using_modules | found on the modules page]].

Commonly used module commands:

* <code>module load <module-name></code> – Load the default version of a software package.
* <code>module load <module-name>/<module-version></code> – Load a specific version.
* <code>module purge</code> – Unload all currently loaded modules.
* <code>module avail</code> – List available modules that can be loaded.
* <code>module list</code> – Show currently loaded modules.
* <code>module spider</code> or <code>module spider <module-name></code> – Search for available modules and their versions.

Handy abbreviations are available:

* <code>ml</code> – Equivalent to <code>module list</code>.
* <code>ml <module-name></code> – Equivalent to <code>module load <module-name></code>.

== Tips for Loading Software ==

Properly managing your software environment is key to avoiding conflicts and ensuring reproducibility. Here are some best practices:

* Avoid loading modules in your <code>.bashrc</code> file. Doing so can cause unexpected behavior, particularly in non-interactive environments like batch jobs or remote shells. For more information, see our [[bashrc guidelines|.bashrc guidelines]].

* Instead, load modules manually or from a separate script. This approach gives you more control and helps keep environments clean.

* Load required modules inside your job submission script. This ensures that your job runs with the expected software environment, regardless of your interactive shell settings.

* Be explicit about module versions. Short names like <code>gcc</code> will load the system default (e.g., <code>gcc/12.3</code>), which may change in the future. Specify full versions (e.g., <code>gcc/13.3</code>) for long-term reproducibility.

* Resolve dependencies with <code>module spider</code>. Some modules depend on others. Use <code>module spider <module-name></code> to discover which modules are required and how to load them in the correct order. For more, see [[Using_modules#Module_spider | Using <code>module spider</code>]].

== Using Commercial Software ==

You may be able to use commercial software on Trillium, but there are a few important considerations:

* Bring your own license. You can use commercial software on Trillium if you have a valid license. If the software requires a license server, you can connect to it securely using [[SSH_Tunneling | SSH tunneling]].

* SciNet and the {{Alliance}} do not provide user-specific licenses. Due to the large and diverse user base, we cannot provide licenses for individual or specialized commercial packages.

* Freely available commercial tools. Some widely useful commercial tools are available system-wide, such as compilers, math libraries, debuggers.

* Software not available (unless you bring your own license): tools like [[MATLAB]], Gaussian, and IDL are not provided centrally. If you have your own license, you are welcome to install and use them.

* Open-source alternatives are available. Consider using freely available tools such as [[Python]], [[R]], and Octave, which are well-supported and widely used on the system.

* We're here to help. If you have a valid license and need help installing commercial software, feel free to contact us, we'll assist where possible.

A list of commercial software currently installed on Trillium (for which you must supply a license to use) is available on the [[Commercial_software | Commercial Software page]].

= Technical Details =

== Cooling and Energy Efficiency ==

Trillium is fully direct liquid cooled using warm water (35–40 °C input), resulting in:

* PUE below 1.03 (high energy efficiency)
* Use of closed-loop dry fluid coolers, avoiding evaporative towers and new water usage
* Heat reuse: Trillium supplies excess heat to nearby facilities to minimize climate impact

== Storage System ==

The VAST high-performance file system is comprised of a unified 29 PB NVMe-backed storage pool, with:

* 29 PB effective capacity (deduplicated via VAST)
* 16.7 PB raw flash capacity
* 714 GB/s read bandwidth, 275 GB/s write bandwidth
* 10 million read IOPS, 2 million write IOPS
* POSIX and S3 access protocols under a unified namespace
* 48 C-Boxes and 14 D-Boxes for data services

== Backup and Archive Storage ==

An additional 114 PB HPSS tape-based archive is available for nearline storage:

* Dual-copy archive across geographically separate libraries
* Used for both backup and archival purposes
* Backups are managed using Atempo backup software

= Testing and Debugging =

Before submitting your job to the cluster, it's important to test your code to ensure correctness and determine the resources it requires.

* '''Lightweight tests''' can be run directly on the login nodes. As a rule of thumb, these should:
** Run in under a few minutes
** Use no more than 1–2 GB of memory
** Use only 1–2 CPU cores

* You can also run the [[Parallel Debugging with DDT|DDT]] debugger on the login nodes after loading it with: <code>module load ddt-cpu</code>

* For short tests that exceed login node limits or require dedicated resources, request an interactive debug job using the <code>debugjob</code> command:
<pre>
tri-login01:~$ debugjob --clean N
</pre>
Replace <code>N</code> with the number of nodes (1 to 4). If <code>N=1</code>, you will get 1 hour of interactive time; with <code>N=4</code> (the maximum), you will get 22 minutes.
The <code>--clean</code> flag is optional but recommended, as it starts the session with no modules loaded, better mimicking the clean environment of batch jobs.

* If your test job requires more time than allowed by <code>debugjob</code>, you can request an interactive session from the regular queue using <code>salloc</code>:
<pre>
tri-login01:~$ salloc --nodes=N --time=M:00:00 --x11
</pre>
* <code>N</code> is the number of nodes
* <code>M</code> is the number of hours the job should run
* <code>--x11</code> is required for graphical applications (e.g., when using [[Parallel Debugging with DDT|DDT]] or DDD)

'''Note:''' Jobs submitted with <code>salloc</code> may take longer to start, as they are scheduled like any other batch job. See the [[Testing_With_Graphics|Testing with graphics]] page for more information on graphical testing options.

= Submitting Jobs on the CPU Subcluster =

Once you have compiled and tested your code or workflow on the Trillium login nodes and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. These jobs will run on Trillium's compute nodes, and their execution is managed by the SLURM scheduler.

Trillium uses SLURM as its job scheduler. More advanced details of how to interact with the scheduler can be found on the [[Slurm | Slurm page]].

To submit a job, use the <code>sbatch</code> command on a login node:

<pre>
tri-login01:scratch$ sbatch jobscript.sh
</pre>

This places your job into the queue. It will begin execution on available compute nodes when scheduled. Note: jobs must be submitted from a login node, submitting from datamover nodes is not allowed.

In most cases, you should submit jobs from your <code>$SCRATCH</code> directory, not <code>$HOME</code>, since the home directory is read-only on compute nodes. Output from your jobs must be written to <code>$SCRATCH</code>.

Jobs will run under your group's RRG allocation, or, if one is not available, under a RAS allocation (previously called the “default” allocation).

Some example job scripts are shown below.

=== Key Points to Remember ===

* Scheduling is by node, not by core or CPU.
* Each node has 192 cores and 768 GB of memory.
* Jobs are limited to a maximum of 24 hours walltime.
* Output must be written to <code>$SCRATCH</code>. <code>$HOME</code> and <code>$PROJECT</code> are read-only on compute nodes.
* Compute nodes have no internet access.
* Your job script must load all necessary modules explicitly using <code>module load</code>.
* Ensure [[Data_Management#Moving_data|your input data is on Trillium]] before submitting jobs.

== Scheduling by Node ==

On many systems that use SLURM, the scheduler will deduce from the specifications of the number of tasks and the number of CPUs per node what resources should be allocated. On Trillium, things are a bit different.

* All job resource requests on Trillium are scheduled as a multiple of '''nodes'''.
* The nodes that your jobs run on are exclusively yours, for as long as the job is running on them:
** no other user can run jobs on them;
** you can [[SSH]] into your nodes during execution to monitor progress.
* Even if your job does not use all 192 cores, you still get the '''full''' node. Trillium does not share nodes between users.
* Memory requests are ignored. Your job receives <code>N × 768GB </code> of RAM, where <code>N</code> is the number of nodes and 768GB is the amount of memory on each node.
* If running serial or low-core jobs you must still use all 192 cores on the node by bundling multiple independent tasks in one job script. See [[Running_Serial_Jobs_on_Niagara|this page]] for examples.
* If your job underutilizes the cores, our support team may reach out to assist you in optimizing your workflow, or you can [mailto:support@scinet.utoronto.ca contact us] to get assistance.

== Limits ==

There are limits to the size and duration of your jobs, the number of jobs you can run, and the number of jobs you can have queued. It matters whether a user is part of a group with a [https://www.alliancecan.ca/research-portal/accessing-resources/resource-allocation-competitions/ Resources for Research Group allocation] or not. It also matters in which "partition" the job runs. "Partitions" are SLURM-speak for use cases. You specify the partition with the <code>-p</code> parameter to <code>sbatch</code> or <code>salloc</code>, but if you do not specify one, your job will run in the <code>compute</code> partition, which is the most common case.

{| class="wikitable"
!Usage
!Partition
!Limit on Running jobs
!Limit on Submitted jobs (incl. running)
!Min. size of jobs
!Max. size of jobs
!Min. walltime
!Max. walltime
|-
|Compute jobs ||compute || 50 || 1000 || 1 node (192 cores) || default: 20 nodes (3840 cores) with allocation: 1000 nodes (192000 cores)|| 15 minutes || 24 hours
|-
|Testing or troubleshooting || debug || 1 || 1 || 1 node (192 cores) || 4 nodes (768 cores)|| N/A || 1 hour
|-
|Archiving or retrieving data in [[HPSS]]|| archivelong || 2 per user (5 in total) || 10 per user || N/A || N/A|| 15 minutes || 72 hours
|-
|Inspecting archived data, small archival actions in [[HPSS]] || archiveshort vfsshort || 2 per user|| 10 per user || N/A || N/A || 15 minutes || 1 hour
|}

Even if you respect these limits, your jobs will still have to wait in the queue. The waiting time depends on many factors such as your group's allocation amount, how much allocation has been used in the recent past, the number of requested nodes and walltime, and how many other jobs are waiting in the queue.

== Example submission script (MPI) ==

<source lang="bash">
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=192
#SBATCH --time=01:00:00
#SBATCH --job-name=mpi_job
#SBATCH --output=mpi_output_%j.txt
#SBATCH --mail-type=FAIL

cd $SLURM_SUBMIT_DIR

module load StdEnv/2023
module load gcc/12.3
module load openmpi/4.1.5

mpirun ./mpi_example
# or "srun ./mpi_example"
</source>

Submit this script from your <code>$SCRATCH</code> directory with the command:

<pre>
tri-login01:scratch$ sbatch mpi_job.sh
</pre>

<ul>
<li>First line indicates that this is a bash script.</li>
<li>Lines starting with <code>#SBATCH</code> go to SLURM.</li>
<li><code>sbatch</code> reads these lines as a job request (which it gives the name <code>mpi_job</code>).</li>
<li>In this case, SLURM looks for 2 nodes each running 192 tasks, for 1 hour.</li>
<li>Note that the mpifun flag <code>--ppn</code> (processors per node) is ignored.</li>
<li>Once it finds such a node, it runs the script:
<ul>
<li>Change to the submission directory;</li>
<li>Loads modules;</li>
<li>Runs the <code>mpi_example</code> application (SLURM will inform <code>mpirun</code> or <code>srun</code> how many processes to run).</li>
</ul>
</li>
</ul>

== Example submission script (OpenMP) ==

<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=192
#SBATCH --time=01:00:00
#SBATCH --job-name=openmp_job
#SBATCH --output=openmp_output_%j.txt
#SBATCH --mail-type=FAIL

cd $SLURM_SUBMIT_DIR

module load StdEnv/2023
module load gcc/12.3

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

./openmp_example
# or "srun ./openmp_example"
</source>

Submit this script from your <code>$SCRATCH</code> directory with the command:

<pre>
tri-login01:scratch$ sbatch openmp_job.sh
</pre>

<ul>
<li>First line indicates that this is a Bash script.</li>
<li>Lines starting with <code>#SBATCH</code> are directives for SLURM.</li>
<li><code>sbatch</code> reads these lines as a job request (which it gives the name <code>openmp_job</code>).</li>
<li>In this case, SLURM looks for one node with 192 CPUs for a single task running up to 192 OpenMP threads, for 1 hour.</li>
<li>Once such a node is allocated, it runs the script:
<ul>
<li>Changes to the submission directory;</li>
<li>Loads the required modules;</li>
<li>Sets <code>OMP_NUM_THREADS</code> based on SLURM’s CPU allocation;</li>
<li>Runs the <code>openmp_example</code> application.</li>
</ul>
</li>
</ul>

== Monitoring Queued and Running Jobs ==

Once your job is submitted to the queue, you can monitor its status and performance using the following SLURM commands:

<ul>
<li><code>squeue</code> shows all jobs in the queue. Use <code>squeue -u $USER</code> to view only your jobs.</li>

<li><code>squeue -j JOBID</code> shows the current status of a specific job. Alternatively, use <code>scontrol show job JOBID</code> for detailed information, including allocated nodes, resources, and job flags.
</li>

<li><code>squeue --start -j JOBID</code> gives a rough estimate of when a pending job is expected to start. Note that this estimate is often inaccurate and can change depending on system load and priorities.</li>

<li><code>scancel JOBID</code> cancels a job you submitted.</li>

<li><code>jobperf JOBID</code> gives a live snapshot of the CPU and memory usage of your job while it is running.</li>

<li><code>sacct</code> shows information about your past jobs, including start time, run time, node usage, and exit status.</li>
</ul>

More details on monitoring jobs can be found on the [[Slurm#Monitoring_jobs | Slurm page]].

You can also view and manage your current and past jobs, resource usage, and allocation history through the [https://my.scinet.utoronto.ca my.SciNet] portal.

Trillium Quickstart

2025-08-07T21:24:36Z

Ymeiron: Added GPU login node address

{{Infobox Computer
|image=[[Image:Trillium.jpg|center|300px|thumb]]
|name=Trillium
|installed=Aug 2025
|operatingsystem= Rocky Linux 9.6
|loginnode= trillium.scinet.utoronto.ca, trillium-gpu.scinet.utoronto.ca
|nnodes= 1284 nodes (240768 cores)
|rampernode= 768 GB
|corespernode= 192 (CPU nodes) and 96 (GPU nodes)
|interconnect=Mellanox Dragonfly+
|queuetype=Slurm
}}

__TOC__

= System Overview =

The Trillium system is a state-of-the-art high performance computing platform, consisting of three main components:

1. CPU Subcluster
* ~240,000 cores across homogeneous CPU nodes
* Non-blocking 400 Gb/s NDR InfiniBand interconnect
* Ideal for large-scale parallel workloads

2. GPU Subcluster
* 61 GPU nodes, each with 4 x NVIDIA H100 (SXM) GPUs
* 800 Gb/s bandwidth per node (200 Gb/s per GPU) over InfiniBand
* Optimized for AI/ML and accelerated science workloads
* Note: This subcluster is in high demand and not ideal for training extremely large models (multi-100B parameters)

3. Storage System
* Unified 29 PB VAST NVMe storage for all workloads
* No tiering — all flash-based for consistent performance
* Accessible via POSIX or S3 under a unified namespace

== Specifications ==

The Trillium cluster is a large cluster comprised of two types of nodes:

{| class="wikitable sortable"
! nodes !! cores !! available memory !! CPU !! GPU
|-
| 1224 || 192 || 768GB DDR5 ||2 x AMD EPYC 9655 (Zen 5) @ 2.6 GHz, 384MB cache L3 ||
|-
| 60 || 96 || 768GB DDR5 || 1 x AMD EPYC 9654 (Zen 4) @ 2.4 GHz, 384MB cache L3 || 4 x NVIDIA H100 SXM (80 GB memory)
|}

Each node of the cluster has 768 GB RAM per node. Being designed for large parallel workloads, it has a fast interconnect consisting of NDR InfiniBand in a Dragonfly+ topology with Adaptive Routing. The compute nodes are accessed through a queueing system that allows jobs with a minimum of 15 minutes and a maximum of 24 hours.

== Storage System ==

Trillium features a unified high-performance storage system based on the VAST platform, with no tiering. It serves the following directories:

* <code>/home</code> – For personal files and configurations.
* <code>/scratch</code> – High-speed, temporary storage for job data.
* <code>/project</code> – Shared storage for project teams and collaborations.

The storage is accessible via the NDR InfiniBand fabric for maximum performance across all workloads.

= Getting started on Trillium =

Access to Trillium is not enabled automatically for everyone with an account with the {{DigitalResearchAllianceOfCanada}}, but anyone with an active Alliance account can get their access enabled.

Trillium is not automatically available to all Alliance account holders. If you are new to SciNet or your Supervisor/PI does not hold a current {{Alliance}} [https://alliancecan.ca/en/services/advanced-research-computing/research-portal/accessing-resources/resource-allocation-competitions RAC] allocation, you will need to request access on the [https://ccdb.alliancecan.ca/me/access_systems Access Systems] page on the CCDB site. After clicking the "I request access" button, it usually takes only one or two business days for access to be granted.

You can check if you already have Trillium access by attempting to log in. If you receive a "Permission denied" error (and your SSH key is correctly set up), you may need to opt in.

Please read this document carefully. The [https://docs.scinet.utoronto.ca/index.php/FAQ FAQ] is also a useful resource. If at any time you require assistance, or if something is unclear, please do not hesitate to [mailto:support@scinet.utoronto.ca contact us].

== Logging in ==

Trillium runs Rocky Linux 9.6, which is a type of Linux. You will need to be familiar with Linux systems to work on Trillium. If you are not it will be worth your time to review our [https://support.scinet.utoronto.ca/education/browse.php?category=-1&search=scmp101&include=all&filter=Filter Introduction to Linux Shell] class.

As with all SciNet and {{Alliance}} compute systems, access to Trillium is done via [[SSH]] (secure shell) only and authentication is only allowed via SSH keys. [https://docs.alliancecan.ca/wiki/SSH_Keys Please refer to this page] to generate your SSH key pair and make sure you use them securely.

Open a terminal window (e.g. Connecting with [https://docs.alliancecan.ca/wiki/Connecting_with_PuTTY PuTTY] on Windows or Connecting with [https://docs.alliancecan.ca/wiki/Connecting_with_MobaXTerm MobaXTerm]), then SSH into the Trillium login nodes with your {{Alliance}} credentials:

$ ssh -i /path/to/ssh_private_key -Y MYALLIANCEUSERNAME@trillium.scinet.utoronto.ca

* The Trillium login nodes are where you develop, edit, compile, prepare and submit jobs.
* These login nodes are not part of the Trillium compute cluster, but have the same architecture, operating system, and software stack.
* The optional <code>-Y</code> enables X11 forwarding, allowing graphical programs to open windows on your local computer.
* To run on Trillium compute nodes, you must [[#Submitting_jobs | submit a batch job]].

If you cannot log in, be sure to first check the [https://docs.scinet.utoronto.ca System Status] on this site's front page.

Note: We plan to add browser access to Trillium via Open OnDemand in the future. In the meantime you can still access our existing Open OnDemand deployment by following the instructions in our [https://docs.scinet.utoronto.ca/index.php/Open_OnDemand_Quickstart quickstart guide].

== Software Environment ==

Trillium uses the '''environment modules''' system to manage compilers, libraries, and other software packages. Modules dynamically modify your environment (e.g., <code>PATH</code>, <code>LD_LIBRARY_PATH</code>) so you can access different versions of software without conflicts.

A detailed explanation can be [[Using_modules | found on the modules page]].

Commonly used module commands:

* <code>module load <module-name></code> – Load the default version of a software package.
* <code>module load <module-name>/<module-version></code> – Load a specific version.
* <code>module purge</code> – Unload all currently loaded modules.
* <code>module avail</code> – List available modules that can be loaded.
* <code>module list</code> – Show currently loaded modules.
* <code>module spider</code> or <code>module spider <module-name></code> – Search for available modules and their versions.

Handy abbreviations are available:

* <code>ml</code> – Equivalent to <code>module list</code>.
* <code>ml <module-name></code> – Equivalent to <code>module load <module-name></code>.

== Tips for Loading Software ==

Properly managing your software environment is key to avoiding conflicts and ensuring reproducibility. Here are some best practices:

* Avoid loading modules in your <code>.bashrc</code> file. Doing so can cause unexpected behavior, particularly in non-interactive environments like batch jobs or remote shells. For more information, see our [[bashrc guidelines|.bashrc guidelines]].

* Instead, load modules manually or from a separate script. This approach gives you more control and helps keep environments clean.

* Load required modules inside your job submission script. This ensures that your job runs with the expected software environment, regardless of your interactive shell settings.

* Be explicit about module versions. Short names like <code>gcc</code> will load the system default (e.g., <code>gcc/12.3</code>), which may change in the future. Specify full versions (e.g., <code>gcc/13.3</code>) for long-term reproducibility.

* Resolve dependencies with <code>module spider</code>. Some modules depend on others. Use <code>module spider <module-name></code> to discover which modules are required and how to load them in the correct order. For more, see [[Using_modules#Module_spider | Using <code>module spider</code>]].

== Using Commercial Software ==

You may be able to use commercial software on Trillium, but there are a few important considerations:

* Bring your own license. You can use commercial software on Trillium if you have a valid license. If the software requires a license server, you can connect to it securely using [[SSH_Tunneling | SSH tunneling]].

* SciNet and the {{Alliance}} do not provide user-specific licenses. Due to the large and diverse user base, we cannot provide licenses for individual or specialized commercial packages.

* Freely available commercial tools. Some widely useful commercial tools are available system-wide, such as compilers, math libraries, debuggers.

* Software not available (unless you bring your own license): tools like [[MATLAB]], Gaussian, and IDL are not provided centrally. If you have your own license, you are welcome to install and use them.

* Open-source alternatives are available. Consider using freely available tools such as [[Python]], [[R]], and Octave, which are well-supported and widely used on the system.

* We're here to help. If you have a valid license and need help installing commercial software, feel free to contact us, we'll assist where possible.

A list of commercial software currently installed on Trillium (for which you must supply a license to use) is available on the [[Commercial_software | Commercial Software page]].

= Technical Details =

== Cooling and Energy Efficiency ==

Trillium is fully direct liquid cooled using warm water (35–40 °C input), resulting in:

* PUE below 1.03 (high energy efficiency)
* Use of closed-loop dry fluid coolers, avoiding evaporative towers and new water usage
* Heat reuse: Trillium supplies excess heat to nearby facilities to minimize climate impact

== Storage System ==

The VAST high-performance file system is comprised of a unified 29 PB NVMe-backed storage pool, with:

* 29 PB effective capacity (deduplicated via VAST)
* 16.7 PB raw flash capacity
* 714 GB/s read bandwidth, 275 GB/s write bandwidth
* 10 million read IOPS, 2 million write IOPS
* POSIX and S3 access protocols under a unified namespace
* 48 C-Boxes and 14 D-Boxes for data services

== Backup and Archive Storage ==

An additional 114 PB HPSS tape-based archive is available for nearline storage:

* Dual-copy archive across geographically separate libraries
* Used for both backup and archival purposes
* Backups are managed using Atempo backup software

= Testing and Debugging =

Before submitting your job to the cluster, it's important to test your code to ensure correctness and determine the resources it requires.

* '''Lightweight tests''' can be run directly on the login nodes. As a rule of thumb, these should:
** Run in under a few minutes
** Use no more than 1–2 GB of memory
** Use only 1–2 CPU cores

* You can also run the [[Parallel Debugging with DDT|DDT]] debugger on the login nodes after loading it with: <code>module load ddt-cpu</code>

* For short tests that exceed login node limits or require dedicated resources, request an interactive debug job using the <code>debugjob</code> command:
<pre>
tri-login01:~$ debugjob --clean N
</pre>
Replace <code>N</code> with the number of nodes (1 to 4). If <code>N=1</code>, you will get 1 hour of interactive time; with <code>N=4</code> (the maximum), you will get 22 minutes.
The <code>--clean</code> flag is optional but recommended, as it starts the session with no modules loaded, better mimicking the clean environment of batch jobs.

* If your test job requires more time than allowed by <code>debugjob</code>, you can request an interactive session from the regular queue using <code>salloc</code>:
<pre>
tri-login01:~$ salloc --nodes=N --time=M:00:00 --x11
</pre>
* <code>N</code> is the number of nodes
* <code>M</code> is the number of hours the job should run
* <code>--x11</code> is required for graphical applications (e.g., when using [[Parallel Debugging with DDT|DDT]] or DDD)

'''Note:''' Jobs submitted with <code>salloc</code> may take longer to start, as they are scheduled like any other batch job. See the [[Testing_With_Graphics|Testing with graphics]] page for more information on graphical testing options.

= Submitting Jobs on the CPU Subcluster =

Once you have compiled and tested your code or workflow on the Trillium login nodes and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. These jobs will run on Trillium's compute nodes, and their execution is managed by the SLURM scheduler.

Trillium uses SLURM as its job scheduler. More advanced details of how to interact with the scheduler can be found on the [[Slurm | Slurm page]].

To submit a job, use the <code>sbatch</code> command on a login node:

<pre>
tri-login01:scratch$ sbatch jobscript.sh
</pre>

This places your job into the queue. It will begin execution on available compute nodes when scheduled. Note: jobs must be submitted from a login node, submitting from datamover nodes is not allowed.

In most cases, you should submit jobs from your <code>$SCRATCH</code> directory, not <code>$HOME</code>, since the home directory is read-only on compute nodes. Output from your jobs must be written to <code>$SCRATCH</code>.

Jobs will run under your group's RRG allocation, or, if one is not available, under a RAS allocation (previously called the “default” allocation).

Some example job scripts are shown below.

=== Key Points to Remember ===

* Scheduling is by node, not by core or CPU.
* Each node has 192 cores and 768 GB of memory.
* Jobs are limited to a maximum of 24 hours walltime.
* Output must be written to <code>$SCRATCH</code>. <code>$HOME</code> and <code>$PROJECT</code> are read-only on compute nodes.
* Compute nodes have no internet access.
* Your job script must load all necessary modules explicitly using <code>module load</code>.
* Ensure [[Data_Management#Moving_data|your input data is on Trillium]] before submitting jobs.

== Scheduling by Node ==

On many systems that use SLURM, the scheduler will deduce from the specifications of the number of tasks and the number of CPUs per node what resources should be allocated. On Trillium, things are a bit different.

* All job resource requests on Trillium are scheduled as a multiple of '''nodes'''.
* The nodes that your jobs run on are exclusively yours, for as long as the job is running on them:
** no other user can run jobs on them;
** you can [[SSH]] into your nodes during execution to monitor progress.
* Even if your job does not use all 192 cores, you still get the '''full''' node. Trillium does not share nodes between users.
* Memory requests are ignored. Your job receives <code>N × 768GB </code> of RAM, where <code>N</code> is the number of nodes and 768GB is the amount of memory on each node.
* If running serial or low-core jobs you must still use all 192 cores on the node by bundling multiple independent tasks in one job script. See [[Running_Serial_Jobs_on_Niagara|this page]] for examples.
* If your job underutilizes the cores, our support team may reach out to assist you in optimizing your workflow, or you can [mailto:support@scinet.utoronto.ca contact us] to get assistance.

== Limits ==

There are limits to the size and duration of your jobs, the number of jobs you can run, and the number of jobs you can have queued. It matters whether a user is part of a group with a [https://www.alliancecan.ca/research-portal/accessing-resources/resource-allocation-competitions/ Resources for Research Group allocation] or not. It also matters in which "partition" the job runs. "Partitions" are SLURM-speak for use cases. You specify the partition with the <code>-p</code> parameter to <code>sbatch</code> or <code>salloc</code>, but if you do not specify one, your job will run in the <code>compute</code> partition, which is the most common case.

{| class="wikitable"
!Usage
!Partition
!Limit on Running jobs
!Limit on Submitted jobs (incl. running)
!Min. size of jobs
!Max. size of jobs
!Min. walltime
!Max. walltime
|-
|Compute jobs ||compute || 50 || 1000 || 1 node (192 cores) || default: 20 nodes (3840 cores) with allocation: 1000 nodes (192000 cores)|| 15 minutes || 24 hours
|-
|Testing or troubleshooting || debug || 1 || 1 || 1 node (192 cores) || 4 nodes (768 cores)|| N/A || 1 hour
|-
|Archiving or retrieving data in [[HPSS]]|| archivelong || 2 per user (5 in total) || 10 per user || N/A || N/A|| 15 minutes || 72 hours
|-
|Inspecting archived data, small archival actions in [[HPSS]] || archiveshort vfsshort || 2 per user|| 10 per user || N/A || N/A || 15 minutes || 1 hour
|}

Even if you respect these limits, your jobs will still have to wait in the queue. The waiting time depends on many factors such as your group's allocation amount, how much allocation has been used in the recent past, the number of requested nodes and walltime, and how many other jobs are waiting in the queue.

== Example submission script (MPI) ==

<source lang="bash">
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=192
#SBATCH --time=01:00:00
#SBATCH --job-name=mpi_job
#SBATCH --output=mpi_output_%j.txt
#SBATCH --mail-type=FAIL

cd $SLURM_SUBMIT_DIR

module load StdEnv/2023
module load gcc/12.3
module load openmpi/4.1.5

mpirun ./mpi_example
# or "srun ./mpi_example"
</source>

Submit this script from your <code>$SCRATCH</code> directory with the command:

<pre>
tri-login01:scratch$ sbatch mpi_job.sh
</pre>

<ul>
<li>First line indicates that this is a bash script.</li>
<li>Lines starting with <code>#SBATCH</code> go to SLURM.</li>
<li><code>sbatch</code> reads these lines as a job request (which it gives the name <code>mpi_job</code>).</li>
<li>In this case, SLURM looks for 2 nodes each running 192 tasks, for 1 hour.</li>
<li>Note that the mpifun flag <code>--ppn</code> (processors per node) is ignored.</li>
<li>Once it finds such a node, it runs the script:
<ul>
<li>Change to the submission directory;</li>
<li>Loads modules;</li>
<li>Runs the <code>mpi_example</code> application (SLURM will inform <code>mpirun</code> or <code>srun</code> how many processes to run).</li>
</ul>
</li>
</ul>

== Example submission script (OpenMP) ==

<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=192
#SBATCH --time=01:00:00
#SBATCH --job-name=openmp_job
#SBATCH --output=openmp_output_%j.txt
#SBATCH --mail-type=FAIL

cd $SLURM_SUBMIT_DIR

module load StdEnv/2023
module load gcc/12.3

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

./openmp_example
# or "srun ./openmp_example"
</source>

Submit this script from your <code>$SCRATCH</code> directory with the command:

<pre>
tri-login01:scratch$ sbatch openmp_job.sh
</pre>

<ul>
<li>First line indicates that this is a Bash script.</li>
<li>Lines starting with <code>#SBATCH</code> are directives for SLURM.</li>
<li><code>sbatch</code> reads these lines as a job request (which it gives the name <code>openmp_job</code>).</li>
<li>In this case, SLURM looks for one node with 192 CPUs for a single task running up to 192 OpenMP threads, for 1 hour.</li>
<li>Once such a node is allocated, it runs the script:
<ul>
<li>Changes to the submission directory;</li>
<li>Loads the required modules;</li>
<li>Sets <code>OMP_NUM_THREADS</code> based on SLURM’s CPU allocation;</li>
<li>Runs the <code>openmp_example</code> application.</li>
</ul>
</li>
</ul>

== Monitoring Queued and Running Jobs ==

Once your job is submitted to the queue, you can monitor its status and performance using the following SLURM commands:

<ul>
<li><code>squeue</code> shows all jobs in the queue. Use <code>squeue -u $USER</code> to view only your jobs.</li>

<li><code>squeue -j JOBID</code> shows the current status of a specific job. Alternatively, use <code>scontrol show job JOBID</code> for detailed information, including allocated nodes, resources, and job flags.
</li>

<li><code>squeue --start -j JOBID</code> gives a rough estimate of when a pending job is expected to start. Note that this estimate is often inaccurate and can change depending on system load and priorities.</li>

<li><code>scancel JOBID</code> cancels a job you submitted.</li>

<li><code>jobperf JOBID</code> gives a live snapshot of the CPU and memory usage of your job while it is running.</li>

<li><code>sacct</code> shows information about your past jobs, including start time, run time, node usage, and exit status.</li>
</ul>

More details on monitoring jobs can be found on the [[Slurm#Monitoring_jobs | Slurm page]].

You can also view and manage your current and past jobs, resource usage, and allocation history through the [https://my.scinet.utoronto.ca my.SciNet] portal.

Main Page

2025-08-07T21:16:18Z

Ymeiron: Trillium is up!!!

Trillium Quickstart

2025-08-05T20:14:19Z

Ymeiron: Changed the picture

2022-03-09T15:57:31Z

Ymeiron: Updated job limits

{{Infobox Computer
|image=[[Image:Mist.jpg|center|300px|thumb]]
|name=Mist
|installed=Dec 2019
|operatingsystem= Red Hat Enterprise Linux 8.2
|loginnode= mist.scinet.utoronto.ca
|nnodes= 54 IBM AC922
|rampernode= 256 GB
|gpuspernode=4 V100-SMX2-32GB
|interconnect=Mellanox EDR
|vendorcompilers= NVCC, IBM XL
|queuetype=Slurm
}}

=Specifications=
Mist is a SciNet-[[#SOSCIP Users |SOSCIP]] joint GPU cluster consisting of 54 IBM AC922 servers. Each node of the cluster has 32 IBM Power9 cores, 256GB RAM and 4 NVIDIA V100-SMX2-32GB GPU with NVLINKs in between. The cluster has InfiniBand EDR interconnection providing GPU-Direct RMDA capability.

'''Important note:''' the majority of computer systems as of 2021 (laptops, desktops, and HPC) use the 64 bit x86 instruction set architecture (ISA) in their microprocessors produced by Intel and AMD. This ISA is incompatible with Mist, whose hardware uses the 64 bit PPC ISA (set to little endian mode). The practical meaning is that x86-compiled binaries (executables and libraries) cannot be installed on Mist. For this reason, the Niagara and Compute Canada software stacks (modules) cannot be made available on Mist, and using closed-source software is only possible when the vendor provides a compatible version of their application. '''Python applications''' almost always rely on bindings to libraries originally written in C or C++, some of them are not available on PyPI or various Conda channels as precompiled binaries compatible with Mist. The recommended way to use Python on Mist is to create a [[#Anaconda (Python)|Conda]] environment and install packages from the anaconda (default) channel, where most popular packages have a linux-ppc64le (Mist-compatible) version available. Some popular machine learning packages should be installed from the internal [[#Open-CE|Open-CE]] channel. Where a compatible Conda package cannot be found, installing from PyPI (<code>pip install</code>) can be attempted. Pip will attempt to compile the package’s source code if no compatible precompiled wheel is available, therefore a compiler module (such as <code>gcc/.core</code>) should be loaded in advance. Some packages require tweaking of the source code or build procedure to successfully compile on Mist, please contact [[#Support|support]] if you need assistance.

= Getting started on Mist =
As of January 22 2022, authentication is only allowed via SSH keys. [https://docs.computecanada.ca/wiki/SSH_Keys Please refer to this page] to generate your SSH key pair and make sure you use them securely.

Mist can be accessed directly:
<pre>
ssh -i /path/to/ssh_private_key -Y MYCCUSERNAME@mist.scinet.utoronto.ca
</pre>
Mist login node '''mist-login01''' can also be accessed via Niagara cluster.
<pre>
ssh -i /path/to/ssh_private_key -Y MYCCUSERNAME@niagara.scinet.utoronto.ca
ssh -Y mist-login01
</pre>
== Storage ==
The filesystem for Mist is shared with Niagara cluster. See [https://docs.scinet.utoronto.ca/index.php/Niagara_Quickstart#Your_various_directories Niagara Storage] for more details.

= Loading software modules =

You have two options for running code on Mist: use existing software, or compile your own. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]] and a list of [[Modules for Mist]] is also available.

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.
== Tips for loading software ==

* We advise '''''against''''' loading modules in your .bashrc. This can lead to very confusing behaviour under certain circumstances. Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>cuda</code> → <code>cuda/11.0.3</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Modules often require other modules to be loaded first. Solve these dependencies by using [[Using_modules#Module_spider | <code>module spider</code>]].

= Available compilers and interpreters =
* <tt>cuda</tt> module has to be loaded first for GPU software.
* For most compiled software, one should use the GNU compilers (<tt>gcc</tt> for C, <tt>g++</tt> for C++, and <tt>gfortran</tt> for Fortran). Loading <tt>gcc</tt> module makes these available.
* The IBM XL compiler suite (<tt>xlc_r, xlc++_r, xlf_r</tt>) is also available, if you load one of the <tt>xl</tt> modules.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> or <tt>spectrum-mpi</tt> module.

=== CUDA ===

The current installed CUDA Tookits are '''11.0.3''' and '''10.2.2 (10.2.89)'''
<pre>
module load cuda/11.0.3
module load cuda/10.2.2
</pre>
*A compiler (GCC, XL or NVHPC/PGI) module must be loaded in order to use CUDA to build any code.
The current NVIDIA driver version is 450.119.04.

===GNU Compilers ===

Available GCC modules are:
<pre>
gcc/9.3.0 (must load CUDA 11)
gcc/8.5.0 (must load CUDA 10 or 11)
gcc/10.3.0 (w/o CUDA)
</pre>

=== IBM XL Compilers ===

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xl/16.1.1.10
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER9 CPU. Information about the IBM XL Compilers can be found at the following links:[https://www.ibm.com/support/knowledgecenter/SSXVZZ_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++],
[https://www.ibm.com/support/knowledgecenter/SSAT4T_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

=== OpenMPI ===
<tt>openmpi/<version></tt> module is avaiable with different compilers including GCC and XL. <tt>spectrum-mpi/<version></tt> module provides IBM Spectrum MPI.

=== NVHPC/PGI ===
PGI compiler is provided in NVHPC (NVIDIA HPC SDK).
<pre>
module load nvhpc/21.3
</pre>

= Software =
== Amber20 ==

Users who hold Amber20 license can build Amber20 from its source code and run on Mist. '''SOSCIP/SciNet doesn't provide Amber license or source code.'''

=== Building Amber20 ===
Modules that are needed for building Amber20:
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 anaconda3/2021.05 cmake/3.19.8
</pre>
Cmake configuration:
<pre>
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/where-amber-install -DCOMPILER=GNU -DMPI=FALSE -DCUDA=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=FALSE -DOPENMP=TRUE -DNCCL=FALSE -DAPPLY_UPDATES=TRUE
</pre>

=== Running Amber20 ===
'''NVIDIA Pascal P100 and later GPUs like V100 do not scale beyond a single GPU'''. It is highly suggested to run Amber20 as a single-gpu job.
A job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP-project-ID>

module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 anaconda3/2021.05
export PATH=$HOME/where-amber-install/bin:$PATH
export LD_LIBRARY_PATH=$HOME/where-amber-install/lib:$LD_LIBRARY_PATH
pmemd.cuda .... <parameters> ...
</pre>

== Anaconda (Python) ==
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation. Anaconda is provided as modules: '''anaconda3'''

To install Anaconda locally, user need to load the module and create a conda environment:
<pre>
module load anaconda3
conda create -n myPythonEnv python=3.8
</pre>
*Note: By default, conda environments are located in '''$HOME/.conda/envs'''. Cache (downloaded tarballs and packages) is under '''$HOME/.conda/pkgs'''. User may run into problem with disk quota if there are too many environments created. To clean conda cache, '''please run: "conda clean -y --all" and "rm -rf $HOME/.conda/pkgs/*" after installation of packages'''.

To activate the conda environment: (should be activated before running python)
<pre>
source activate myPythonEnv
</pre>
Note that you SHOULD NOT use '''conda activate myPythonEnv''' to activate the environment. This leads to all sorts of problems. Once the environment is activated, user can update or install packages via '''conda''' or '''pip'''
<pre>
conda install <package_name> (preferred way to install packages)
pip install <package_name>
</pre>
*Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>
To deactivate:
<pre>
source deactivate
</pre>
To remove a conda environment:
<pre>
conda remove --name myPythonEnv --all
</pre>
To verify that the environment was removed, run:
<pre>
conda info --envs
</pre>

=== Submitting Python Job ===
A single-gpu job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate myPythonEnv
python code.py ...
</pre>

== CuPy ==
[https://cupy.chainer.org CuPy] is an open-source matrix library accelerated with NVIDIA CUDA. It also uses CUDA-related libraries including cuBLAS, cuDNN, cuRand, cuSolver, cuSPARSE, cuFFT and NCCL to make full use of the GPU architecture. CuPy is an implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it. It supports a subset of numpy.ndarray interface.

CuPy can be install into any conda environment. Python packages: numpy, six and fastrlock are required. cuDNN and NCCL are optional.
<pre>
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 nccl/2.9.9 anaconda3/2021.05
conda create -n cupy-env python=3.8 numpy six fastrlock
source activate cupy-env
CFLAGS="-I$MODULE_CUDNN_PREFIX/include -I$MODULE_NCCL_PREFIX/include -I$MODULE_CUDA_PREFIX/include" LDFLAGS="-L$MODULE_CUDNN_PREFIX/lib64 -L$MODULE_NCCL_PREFIX/lib" CUDA_PATH=$MODULE_CUDA_PREFIX pip install cupy
#building/installing CuPy will take a few minutes
</pre>

== Gromacs ==
[http://www.gromacs.org/ GROMACS] is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.
*'''GROMACS 2019'''
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2019.6
</pre>
*'''GROMACS 2020 and 2021''' Thread-MPI version supports full GPU enablement of all key computational sections. The GPU is used throughout the timestep and repeated CPU-GPU transfers are eliminated. Users are suggested to carefully verify the results.
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2020.4
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2020.6
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.2
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 openmpi/4.1.1+ucx-1.10.0 gromacs/2021.2 (testing purpose only)
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.4
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.5
</pre>
=== Small/Medium Simulation ===
Due to the lack of PME domain decomposition support on GPU, Gromacs uses CPU to calculate PME when using multiple GPUs. '''It is always recommended to use a single GPU to do small and medium sized simulations with Gromacs.''' By using only 1 tMPI thread (w/ multiple OpenMP threads) on a single GPU, both non-bonded PP and PME are atomically offloaded to GPU when possible.
* Gromacs 2019 example:
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1

module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2019.6
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
gmx mdrun -pin off -ntmpi 1 -ntomp 8 ... <other parameters>
</pre>

* Gromacs 2020 or 2021 example:
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.5
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
export GMX_FORCE_UPDATE_DEFAULT_GPU=true
gmx mdrun -pin off -ntmpi 1 -ntomp 8 ... <other parameters>
</pre>

=== Large Simulation ===
If memory size (~58GB) for single-gpu job is not sufficient for the simulation, multiple GPUs can be used. It is suggested to test starting with one full node with 4GPUs and force PME on GPU. Multiple PME ranks are not supported with PME on GPU, so if GPU is used for the PME calculation -npme (number of PME ranks) must be set to 1. If PME has less work than PP, it is suggested to run multiple ranks per GPU, so the GPU for PME rank can also do some work on PP rank(s).
'''If your simulation can fit in a single GPU job, please use single GPU to get much higher efficiency. Do not waste 3 additional GPU resource for getting only a small performance improvement.
'''
*An example using 4 GPUs, 7 PP ranks/tmpi threads + 1 PME rank/tmpi thread: ('''-pin on -pme gpu -npme 1''' must be added to mdrun command in order to force GPU to do PME)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.5

export OMP_NUM_THREADS=4
gmx mdrun -ntmpi 8 -pin on -pme gpu -npme 1 ... <add your parameters>
</pre>
*It is suggested to also test using '''-ntmpi 4''' and '''export OMP_NUM_THREADS=8''' if you receive a NOTE in Gromacs output saying "% performance was lost because the PME ranks had more work to do than the PP ranks". In this case, NVIDIA MPS is not needed since there is only one MPI rank per GPU.
*'''Please note that the solving of PME on GPU is still only the initial version supporting this behaviour, and comes with a set of limitations outlined further below.'''
<pre>
* Only a PME order of 4 is supported on GPUs.
* PME will run on a GPU only when exactly one rank has a PME task, ie. decompositions with multiple ranks doing PME are not supported.
* Only single precision is supported.
* Free energy calculations where charges are perturbed are not supported, because only single PME grids can be calculated.
* Only dynamical integrators are supported (ie. leap-frog, Velocity Verlet, stochastic dynamics)
* LJ PME is not supported on GPUs.
</pre>
*An example using 4 GPUs, '''PME on CPU''': ('''-pin on''' must be added to mdrun command for proper CPU thread bindings)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.5

export OMP_NUM_THREADS=4
gmx mdrun -ntmpi 8 -pin on ... <add your parameters>

# "-ntmpi 16, OMP_NUM_THREADS=2" and "-ntmpi 4, OMP_NUM_THREADS=8" should also be tested.
# num_thread_MPI_ranks(-ntmpi) * num_OpenMP_threads = 32
</pre>
*'''If your simulation can fit in a single GPU job, please use single GPU to get much higher efficiency. Do not waste 3 additional GPU resource for getting only a small performance improvement.'''
*'''NOTE: The above examples will NOT work with multiple nodes. If simulation is too large for a single GPU node, please contact SciNet/SOSCIP support.'''

== NAMD ==
[http://www.ks.uiuc.edu/Research/namd/ NAMD] is a parallel, object-oriented molecular dynamics code designed for high-performance simulation of large biomolecular systems.
=== 2.14 ===
<pre>
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
</pre>
==== Running with single GPU ====
If you have many jobs to run, it is always suggested to run with a single gpu per job. This makes jobs easier to be scheduled and gives better overall performance.
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=1
#SBATCH --nodes=1

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -bind-to none -hostfile nodelist-$SLURM_JOB_ID `which namd2` +idlepoll +ppn 8 +p 8 stmv.namd
</pre>

==== Running with one process per node (4 GPUs)====
An example of the job script (using 1 node, '''one process per node''', 32 CPU threads per process + 4 GPUs per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 32 +p $((32*SLURM_NTASKS)) stmv.namd
</pre>
==== Running with one process per GPU (4 GPUs)====
NAMD may scale better if using '''one process per GPU'''. Please do your own benchmark.
An example of the job script (using 1 node, '''one process per GPU''', 8 CPU threads per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 4 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 8 +p $((8*SLURM_NTASKS)) stmv.namd
</pre>

== Open-CE ==
[https://github.com/open-ce/open-ce Open-CE] is an '''IBM''' repo for feedstock collection, environment data, and scripts for building Tensorflow, Pytorch, and other machine learning packages and dependencies. Open-CE is distributed as a '''conda channel''' on Mist cluster.
'''Available packages and versions are listed here [https://github.com/open-ce/open-ce/releases/tag/open-ce-v1.5.2 Open-CE Releases]'''. Currently only python 3.8 and CUDA 11.2 are supported. If you need a different python or cuda version, please contact SOSCIP or SciNet support.

*Packages can be installed by setting Open-CE conda channel:
<pre>
conda install -c /scinet/mist/ibm/open-ce python=3.8 cudatoolkit=11.2 PACKAGE
</pre>
*Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

{| class="wikitable"
|+Available Packages:
|-
|Tensorflow
|TensorFlow Estimators
|TensorFlow Probability
|TensorBoard
|TensorBoard Data Server
|TensorFlow Text
|TensorFlow Model Optimizations
|TensorFlow Addons
|TensorFlow Datasets
|TensorFlow Hub
|-
|TensorFlow MetaData
|PyTorch
|TorchText
|TorchVision
|PyTorch Lightning
|PyTorch Lightning Bolts
|ONNX
|Onnx-runtime
|skl2onnx
|tf2onnx
|-
|onnxmltools
|onnxconverter-common
|XGBoost
|LightGBM
|Transformers
|Tokenizers
|SentencePiece
|Spacy
|DALI
|OpenCV
|-
|Horovod
|PyArrow
|grpc
|uwsgi
|ORC
|Mamba
|}

== PyTorch ==
=== Installing from IBM Open-CE Conda Channel ===
The easiest way to install PyTorch on Mist is using IBM's Conda channel. User needs to prepare a conda environment and install PyTorch using IBM's Open-CE Conda channel.
<pre>
module load anaconda3
conda create -n pytorch_env python=3.8
source activate pytorch_env
conda install -c /scinet/mist/ibm/open-ce pytorch=1.10.2 cudatoolkit=11.2
or
conda install -c /scinet/mist/ibm/open-ce-1.2 pytorch=1.7.1 cudatoolkit=11.0 (or 10.2)
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>
Add below command into your job script before python command to get deterministic results, see details here: [https://github.com/pytorch/pytorch/issues/39849]
<pre>
export CUBLAS_WORKSPACE_CONFIG=:4096:2
</pre>

== RAPIDS ==
The [https://rapids.ai RAPIDS] is a suite of open source software libraries that gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. The RAPIDS data science framework includes a collection of libraries: '''cuDF(GPU DataFrames)''', '''cuML(GPU Machine Learning Algorithms)''', '''cuStrings(GPU String Manipulation)''', etc.

=== Installing from IBM Conda Channel ===
The easiest way to install RAPIDS on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install powerai-rapids using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n rapids_env python=3.7
source activate rapids_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/ powerai-rapids
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== TensorFlow and Keras ==
=== Installing from IBM Conda Channel ===
The easiest way to install TensorFlow and Keras on Mist is using IBM's Open-CE Conda channel. User needs to prepare a conda environment and install TensorFlow using IBM's Open-CE Conda channel.
<pre>
module load anaconda3
conda create -n tf_env python=3.8
source activate tf_env
conda install -c /scinet/mist/ibm/open-ce tensorflow==2.7.1 cudatoolkit=11.2
or
conda install -c /scinet/mist/ibm/open-ce-1.2 tensorflow==2.4.3 cudatoolkit=11.0 (or 10.2)
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

= Testing and debugging =
You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.
* Small test jobs can be run on the login node. Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than one gpu and a few cores.

* Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
mist-login01:~$ debugjob --clean -g G
where G is the number of gpus, If G=1, this gives an interactive session for 2 hours, whereas G=4 gets you a single node with 4 gpus for 30 minutes, and with G=8 (the maximum) gets you 2 nodes each with 4 gpus for 15 minutes. The <tt>--clean</tt> argument is optional but recommended as it will start the session without any modules loaded, thus mimicking more closely what happens when you submit a job script. Users needs to load module and activate the conda environment after a debug job starts. It is recommended to do a 'conda clean' before 'source activate ENV' in a debug job if --clean flag is missed.

= Submitting jobs =
Once you have compiled and tested your code or workflow on the Mist login nodes, and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. Your jobs will run on some of Mist's 53 compute nodes. When and where your job runs is determined by the scheduler.

Mist uses SLURM as its job scheduler. It is configured to allow only '''Single-GPU jobs''' and '''Full-node jobs (4 GPUs per node)'''.

You submit jobs from a login node by passing a script to the sbatch command:

mist-login01:scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course. In most cases, you should not submit from your $HOME directory, but rather, from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

Example job scripts can be found below.
Keep in mind:
* Scheduling is by single gpu or by full node, so you ask only 1 gpu or 4 gpus per node.
* Your job's maximum walltime is 24 hours.
* Jobs must write their output to your scratch or project directory (home is read-only on compute nodes).
* Compute nodes have no internet access.
* Your job script will not remember the modules you have loaded, so it needs to contain "module load" commands of all the required modules (see examples below).
== SOSCIP Users ==
*[https://www.soscip.org SOSCIP] is a consortium to bring together industrial partners and academic researchers and provide them with sophisticated advanced computing technologies and expertise to solve social, technical and business challenges across sectors and drive economic growth.

If you are working on a SOSCIP project, please contact [mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca] to have your user account added to SOSCIP project accounts. SOSCIP users need to submit jobs with additional SLURM flag to get higher priority:
<pre>
#SBATCH -A soscip-<SOSCIP_PROJECT_ID> #e.g. soscip-3-001
OR
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID>
</pre>

== Single-GPU job script ==
For a single GPU job, each will have a quarter of the node which is 1 GPU + 8/32 CPU Cores/Threads + ~58GB CPU memory. '''Users should never ask CPU or Memory explicitly.''' If running MPI program, user can set --ntasks to be the number of MPI ranks. '''Do NOT set --ntasks for non-MPI programs.'''
*It is suggested to use NVIDIA Multi-Process Service (MPS) if running multiple MPI ranks on one GPU.
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate conda_env
python code.py ...
</pre>

== Full-node job script ==
'''If you are not sure the program can be executed on multiple GPUs, please follow the single-gpu job instruction above or contact SciNet/SOSCIP support.'''

Multi-GPU job should ask for a minimum of one full node (4 GPUs). User need to specify "compute_full_node" partition in order to get all resource on a node.
*An example for a 1-node job:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4 #this only affects MPI job
#SBATCH --time=1:00:00
#SBATCH -p compute_full_node
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load <modules you need>
Run your program
</pre>

== Limits ==

There are limits to the size and duration of your jobs, the number of jobs you can run and the number of jobs you can have queued.

{| class="wikitable"
!Usage
!Partition
!Running jobs
!Jobs in queue
!Min. size of jobs
!Max. size of jobs
!Min. walltime
!Max. walltime
|-
|Compute jobs ||compute || 100 GPUs || 1000 || 1 GPU (8 cores) || default: 4 nodes (16 GPUs) with allocation: 4 nodes (16 GPUs)|| 15 minutes || 24 hours
|-
|Testing or troubleshooting || debug || 1 || 1 || 1 GPU (8 cores) || 2 nodes (8 GPUs)|| N/A || 2/ngpu hours
|-
|}

Even if you respect these limits, your jobs will still have to wait in the queue. The waiting time depends on many factors such as your group's allocation amount, how much allocation has been used in the recent past, the number of requested nodes and walltime, and how many other jobs are waiting in the queue.

= Jupyter Notebooks =
SciNet’s [[Jupyter Hub]] is a Niagara-type node; it has a different CPU architecture and no GPUs. Conda environments prepared on Mist will not work there properly. Users who need to use Jupyter Notebook to develop and test some aspects of their workflow can create their own server on the Mist login node and use an SSH tunnel to connect to it from outside. Users who choose to do so have to keep in mind that the login node is a shared resource, and heavy calculations should be done only on compute nodes. Processes (including iPython kernels used by the notebooks) are limited to one hour of total CPU time: idle time will not be counted toward this one hour, and use of multiple cores will count proportionally to the number of cores (i.e. a kernel using all 128 virtual cores on the node will be killed after 28 seconds). Idle notebooks can still burden the node by hogging system and GPU memory, please be mindful of other users and terminate notebooks when work is done.

As an example, let us create a new Conda environment and activate it:
<pre>
module load anaconda3
conda create -n jupyter_env python=3.7
source activate jupyter_env
</pre>
Install the Jupyter Notebook server:
<pre>
conda install notebook
</pre>

== Running the notebook server ==
When the Conda environment is active, enter:
<pre>
jupyter-notebook
</pre>
By default, the Jupyter Notebook server uses port 8888 (can be overridden with the <code>--port</code> option). If another user has already started their own server, the default port may be busy, in which case the server will be listening on a different port. Once launched, the server will output some information to the terminal that will include the actual port number used and a 48-character token. For example:
<pre>http://localhost:8890/?token=54c4090d……</pre>
In this example, the server is listening on port 8890.

== Creating a tunnel ==
In order to access this port remotely (i.e. from your office or home), an [https://en.wikipedia.org/wiki/Tunneling_protocol#Secure_Shell_tunneling SSH tunnel] has to be established. Please refer to your SSH client’s documentation for instructions on how to do that. For the OpenSSH client (standard in most Linux distributions and macOS), a tunnel can be opened in a separate terminal session to the one where the Jupyter Notebook server is running. In the new terminal, issue this command:
<pre>
ssh -L8888:localhost:8890 <username>@mist.scinet.utoronto.ca
</pre>
(replace <code><username></code> with your actual username) The tunnel is open as long as this SSH connection is alive. In this example, we tunnel Mist login node’s port 8890 (where our server is assumed to be running) to our home computer’s port 8888 (any other free port is fine). The notebook can be accessed in the browser at the <code><nowiki>http://localhost:8888</nowiki></code> address (followed by <code>/?token=54c4090d……</code>, or the token can be input on the webpage).

== Using Jupyter on compute nodes ==

You can use the instructions here to set up a Jupyter Notebook server on a compute node (including a [[#Testing_and_debugging|debugjob]]). '''We strongly discourage''' you from running an interactive notebook on a compute node (other than for a debugjob), scheduled jobs run in arbitrary times and are not meant to be interactive. Jupyter notebooks can be run non-interactively or converted to Python scripts.

To launch the Jupyter Notebook server, load the <code>anaconda3</code> module and activate your environment as before (by adding the appropriate lines to the submission script, if you are not using the compute node with an interactive shell). Launching the server has to be done like so:
<pre>
HOME=/dev/shm/$USER jupyter-notebook
</pre>
That is because Jupyter will fail unless it can write to the home folder, which is read-only from compute nodes. This modification of the <code>$HOME</code> environment variable will carry over into the notebooks, which is usually not a problem, but in case the notebook relies on this environment variable (e.g. to read certain files), it can be reset manually in the notebook (<code>import os; os.environ['HOME']=……</code>).

Because compute nodes are not accessible from the Internet, tunneling has to be done twice, once from the remote location (office or home) to the Mist login node, and then from the login node to the compute node. Assuming the server is running on port 8890 of the mist006 node, open the first tunnel in a new terminal session in the remote computer:
<pre>
ssh -L8888:localhost:9999 <username>@mist.scinet.utoronto.ca
</pre>
where 9999 is any available port on the Mist login node (to test port availability enter <code>ss -Hln src :9999</code> in the terminal when connected to the Mist login node; an empty output indicates that the port is free). In the same session in the login node that was created with the above command, open the second tunnel to the compute node:
<pre>
ssh -L9999:localhost:8890 mist006
</pre>
Be aware that the second tunnel will automatically disconnect once the job on the compute node times out or is relinquished. The Jupyter Notebook server running on the compute node can now be accessed from the browser as in the previous subsection.

= Support =

SciNet inquiries:
* [mailto:support@scinet.utoronto.ca support@scinet.utoronto.ca]

SOSCIP inquiries:
*[mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca]

Main Page

2022-02-15T15:22:48Z

Ymeiron:

Main Page

2022-02-12T17:59:22Z

Ymeiron:

Database server

2022-01-10T15:22:37Z

Mist

2021-08-09T14:37:44Z

Ymeiron: Note about x86-ppc incompatibility and how to use Python; updated OS version; changed softwares to software

{{Infobox Computer
|image=[[Image:Mist.jpg|center|300px|thumb]]
|name=Mist
|installed=Dec 2019
|operatingsystem= Red Hat Enterprise Linux 8.2
|loginnode= mist.scinet.utoronto.ca
|nnodes= 54 IBM AC922
|rampernode= 256 GB
|gpuspernode=4 V100-SMX2-32GB
|interconnect=Mellanox EDR
|vendorcompilers= NVCC, IBM XL
|queuetype=Slurm
}}

=Specifications=
Mist is a SciNet-[[#SOSCIP Users |SOSCIP]] joint GPU cluster consisting of 54 IBM AC922 servers. Each node of the cluster has 32 IBM Power9 cores, 256GB RAM and 4 NVIDIA V100-SMX2-32GB GPU with NVLINKs in between. The cluster has InfiniBand EDR interconnection providing GPU-Direct RMDA capability.

'''Important note:''' the majority of computer systems as of 2021 (laptops, desktops, and HPC) use the 64 bit x86 instruction set architecture (ISA) in their microprocessors produced by Intel and AMD. This ISA is incompatible with Mist, whose hardware uses the 64 bit PPC ISA (set to little endian mode). The practical meaning is that x86-compiled binaries (executables and libraries) cannot be installed on Mist. For this reason, the Niagara and Compute Canada software stacks (modules) cannot be made available on Mist, and using closed-source software is only possible when the vendor provides a compatible version of their application. '''Python applications''' almost always rely on bindings to libraries originally written in C or C++, some of them are not available on PyPI or various Conda channels as precompiled binaries compatible with Mist. The recommended way to use Python on Mist is to create a [[#Anaconda (Python)|Conda]] environment and install packages from the anaconda (default) channel, where most popular packages have a linux-ppc64le (Mist-compatible) version available. Some popular machine learning packages should be installed from the internal [[#Open-CE|Open-CE]] channel. Where a compatible Conda package cannot be found, installing from PyPI (<code>pip install</code>) can be attempted. Pip will attempt to compile the package’s source code if no compatible precompiled wheel is available, therefore a compiler module (such as <code>gcc/.core</code>) should be loaded in advance. Some packages require tweaking of the source code or build procedure to successfully compile on Mist, please contact [[#Support|support]] if you need assistance.

= Getting started on Mist =
Mist can be accessed directly.
<pre>
ssh -Y MYCCUSERNAME@mist.scinet.utoronto.ca
</pre>
Mist login node '''mist-login01''' can also be accessed via Niagara cluster.
<pre>
ssh -Y MYCCUSERNAME@niagara.scinet.utoronto.ca
ssh -Y mist-login01
</pre>
== Storage ==
The filesystem for Mist is shared with Niagara cluster. See [https://docs.scinet.utoronto.ca/index.php/Niagara_Quickstart#Your_various_directories Niagara Storage] for more details.

= Loading software modules =

You have two options for running code on Mist: use existing software, or compile your own. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]] and a list of [[Modules for Mist]] is also available.

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.
== Tips for loading software ==

* We advise '''''against''''' loading modules in your .bashrc. This can lead to very confusing behaviour under certain circumstances. Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>cuda</code> → <code>cuda/11.0.3</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Modules often require other modules to be loaded first. Solve these dependencies by using [[Using_modules#Module_spider | <code>module spider</code>]].

= Available compilers and interpreters =
* <tt>cuda</tt> module has to be loaded first for GPU software.
* For most compiled software, one should use the GNU compilers (<tt>gcc</tt> for C, <tt>g++</tt> for C++, and <tt>gfortran</tt> for Fortran). Loading <tt>gcc</tt> module makes these available.
* The IBM XL compiler suite (<tt>xlc_r, xlc++_r, xlf_r</tt>) is also available, if you load one of the <tt>xl</tt> modules.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> or <tt>spectrum-mpi</tt> module.

=== CUDA ===

The current installed CUDA Tookits are '''11.0.3''' and '''10.2.2 (10.2.89)'''
<pre>
module load cuda/11.0.3
module load cuda/10.2.2
</pre>
*A compiler (GCC, XL or NVHPC/PGI) module must be loaded in order to use CUDA to build any code.
The current NVIDIA driver version is 450.119.04.

===GNU Compilers ===

Available GCC modules are:
<pre>
gcc/9.3.0 (must load CUDA 11)
gcc/8.5.0 (must load CUDA 10)
gcc/10.3.0 (w/o CUDA)
</pre>

=== IBM XL Compilers ===

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xl/16.1.1.10
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER9 CPU. Information about the IBM XL Compilers can be found at the following links:[https://www.ibm.com/support/knowledgecenter/SSXVZZ_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++],
[https://www.ibm.com/support/knowledgecenter/SSAT4T_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

=== OpenMPI ===
<tt>openmpi/<version></tt> module is avaiable with different compilers including GCC and XL. <tt>spectrum-mpi/<version></tt> module provides IBM Spectrum MPI.

=== NVHPC/PGI ===
PGI compiler is provided in NVHPC (NVIDIA HPC SDK).
<pre>
module load nvhpc/21.3
</pre>

= Software =
== Amber20 ==

Users who hold Amber20 license can build Amber20 from its source code and run on Mist. '''SOSCIP/SciNet doesn't provide Amber license or source code.'''

=== Building Amber20 ===
Modules that are needed for building Amber20:
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 anaconda3/2021.05 cmake/3.19.8
</pre>
Cmake configuration:
<pre>
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/where-amber-install -DCOMPILER=GNU -DMPI=FALSE -DCUDA=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=FALSE -DOPENMP=TRUE -DNCCL=FALSE -DAPPLY_UPDATES=TRUE
</pre>

=== Running Amber20 ===
'''NVIDIA Pascal P100 and later GPUs like V100 do not scale beyond a single GPU'''. It is highly suggested to run Amber20 as a single-gpu job.
A job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP-project-ID>

module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 anaconda3/2021.05
export PATH=$HOME/where-amber-install/bin:$PATH
export LD_LIBRARY_PATH=$HOME/where-amber-install/lib:$LD_LIBRARY_PATH
pmemd.cuda .... <parameters> ...
</pre>

== Anaconda (Python) ==
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation. Anaconda is provided as modules: '''anaconda3'''

To install Anaconda locally, user need to load the module and create a conda environment:
<pre>
module load anaconda3
conda create -n myPythonEnv python=3.8
</pre>
*Note: By default, conda environments are located in '''$HOME/.conda/envs'''. Cache (downloaded tarballs and packages) is under '''$HOME/.conda/pkgs'''. User may run into problem with disk quota if there are too many environments created. To clean conda cache, '''please run: "conda clean -y --all" and "rm -rf $HOME/.conda/pkgs/*" after installation of packages'''.

To activate the conda environment: (should be activated before running python)
<pre>
source activate myPythonEnv
</pre>
Note that you SHOULD NOT use '''conda activate myPythonEnv''' to activate the environment. This leads to all sorts of problems. Once the environment is activated, user can update or install packages via '''conda''' or '''pip'''
<pre>
conda install <package_name> (preferred way to install packages)
pip install <package_name>
</pre>
*Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>
To deactivate:
<pre>
source deactivate
</pre>
To remove a conda environment:
<pre>
conda remove --name myPythonEnv --all
</pre>
To verify that the environment was removed, run:
<pre>
conda info --envs
</pre>

=== Submitting Python Job ===
A single-gpu job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate myPythonEnv
python code.py ...
</pre>

== CuPy ==
[https://cupy.chainer.org CuPy] is an open-source matrix library accelerated with NVIDIA CUDA. It also uses CUDA-related libraries including cuBLAS, cuDNN, cuRand, cuSolver, cuSPARSE, cuFFT and NCCL to make full use of the GPU architecture. CuPy is an implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it. It supports a subset of numpy.ndarray interface.

CuPy can be install into any conda environment. Python packages: numpy, six and fastrlock are required. cuDNN and NCCL are optional.
<pre>
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 nccl/2.9.9 anaconda3/2021.05
conda create -n cupy-env python=3.8 numpy six fastrlock
source activate cupy-env
CFLAGS="-I$MODULE_CUDNN_PREFIX/include -I$MODULE_NCCL_PREFIX/include -I$MODULE_CUDA_PREFIX/include" LDFLAGS="-L$MODULE_CUDNN_PREFIX/lib64 -L$MODULE_NCCL_PREFIX/lib" CUDA_PATH=$MODULE_CUDA_PREFIX pip install cupy
#building/installing CuPy will take a few minutes
</pre>

== Gromacs ==
[http://www.gromacs.org/ GROMACS] is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2019.6
module load MistEnv/2020a cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.6 (old RHEL 7 version for testing only)
</pre>
*'''GROMACS 2020 and 2021''' Thread-MPI version supports full GPU enablement of all key computational sections. The GPU is used throughout the timestep and repeated CPU-GPU transfers are eliminated. Users are suggested to carefully verify the results.
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2020.6
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.2
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 openmpi/4.1.1+ucx-1.10.0 gromacs/2021.2
</pre>
=== Small/Medium Simulation ===
Due to the lack of PME domain decomposition support on GPU, Gromacs uses CPU to calculate PME when using multiple GPUs. '''It is always recommended to use a single GPU to do small and medium sized simulations with Gromacs.''' By using only 1 MPI rank (w/ OpenMP threads) on a single GPU, both non-bonded PP and PME are atomically offloaded to GPU when possible.
* Gromacs 2019 example:
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1

module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2019.6
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
gmx mdrun -pin off -ntmpi 1 -ntomp 8 ... <other parameters>
</pre>

* Gromacs 2020 or 2021 example:
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.2
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
gmx mdrun -pin off -ntmpi 1 -ntomp 8 -update gpu ... <other parameters>
</pre>

=== Large Simulation ===
If memory size (~58GB) for single-gpu job is not sufficient for the simulation, multiple GPUs can be used. It is suggested to test starting with one full node with 4GPUs and force PME on GPU. Multiple PME ranks are not supported with PME on GPU, so if GPU is used for the PME calculation -npme (number of PME ranks) must be set to 1. If PME has less work than PP, it is suggested to run multiple ranks per GPU, so the GPU for PME rank can also do some work on PP rank(s). When running multiple MPI ranks on the same GPU, NVIDIA Multi-Process Service (MPS) must be enabled.
*An example using 4 GPUs, 7 PP ranks + 1 PME rank: ('''-pin on -pme gpu -npme 1''' must be added to mdrun command in order to force GPU to do PME)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 openmpi/4.1.1+ucx-1.10.0 gromacs/2021.2

mkdir -p /dev/shm/nvidia-mps
export CUDA_MPS_PIPE_DIRECTORY=/dev/shm/nvidia-mps
mkdir -p /dev/shm/nvidia-log
export CUDA_MPS_LOG_DIRECTORY=/dev/shm/nvidia-log
nvidia-cuda-mps-control -d

export OMP_NUM_THREADS=4
mpirun -bind-to none gmx_mpi mdrun -pin on -pme gpu -npme 1 ... <add your parameters>
</pre>
*It is suggested to also test using '''--ntasks=4''' and '''OMP_NUM_THREADS=8''' if you receive a NOTE in Gromacs output saying "% performance was lost because the PME ranks had more work to do than the PP ranks". In this case, NVIDIA MPS is not needed since there is only one MPI rank per GPU.
*'''Please note that the solving of PME on GPU is still only the initial version supporting this behaviour, and comes with a set of limitations outlined further below.'''
<pre>
* Only a PME order of 4 is supported on GPUs.
* PME will run on a GPU only when exactly one rank has a PME task, ie. decompositions with multiple ranks doing PME are not supported.
* Only single precision is supported.
* Free energy calculations where charges are perturbed are not supported, because only single PME grids can be calculated.
* Only dynamical integrators are supported (ie. leap-frog, Velocity Verlet, stochastic dynamics)
* LJ PME is not supported on GPUs.
</pre>
*An example using 4 GPUs, '''PME on CPU''': ('''-pin on''' must be added to mdrun command for proper CPU thread bindings)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 openmpi/4.1.1+ucx-1.10.0 gromacs/2021.2

mkdir -p /dev/shm/nvidia-mps
export CUDA_MPS_PIPE_DIRECTORY=/dev/shm/nvidia-mps
mkdir -p /dev/shm/nvidia-log
export CUDA_MPS_LOG_DIRECTORY=/dev/shm/nvidia-log
nvidia-cuda-mps-control -d

export OMP_NUM_THREADS=4
mpirun -bind-to none gmx_mpi mdrun -pin on ... <add your parameters>

# "--ntasks=16, OMP_NUM_THREADS=2" and "--ntasks=4, OMP_NUM_THREADS=8" should also be tested.
# num_Tasks(MPI_ranks) * num_OpenMP_threads = 32
</pre>
*'''NOTE: The above examples will NOT work with multiple nodes. If simulation is too large for a single GPU node, please contact SciNet/SOSCIP support.'''

== NAMD ==
[http://www.ks.uiuc.edu/Research/namd/ NAMD] is a parallel, object-oriented molecular dynamics code designed for high-performance simulation of large biomolecular systems.
=== 2.14 ===
<pre>
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
</pre>
==== Running with single GPU ====
If you have many jobs to run, it is always suggested to run with a single gpu per job. This makes jobs easier to be scheduled and gives better overall performance.
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=1
#SBATCH --nodes=1

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -bind-to none -hostfile nodelist-$SLURM_JOB_ID `which namd2` +idlepoll +ppn 8 +p 8 stmv.namd
</pre>

==== Running with one process per node (4 GPUs)====
An example of the job script (using 1 node, '''one process per node''', 32 CPU threads per process + 4 GPUs per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 32 +p $((32*SLURM_NTASKS)) stmv.namd
</pre>
==== Running with one process per GPU (4 GPUs)====
NAMD may scale better if using '''one process per GPU'''. Please do your own benchmark.
An example of the job script (using 1 node, '''one process per GPU''', 8 CPU threads per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 4 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 8 +p $((8*SLURM_NTASKS)) stmv.namd
</pre>

== Open-CE ==
[https://github.com/open-ce/open-ce Open-CE] is an '''IBM''' repo for feedstock collection, environment data, and scripts for building Tensorflow, Pytorch, XGBoost, and other related packages and dependencies. Open-CE is distributed as a '''conda channel''' on Mist cluster.
Available packages and versions are listed here [https://github.com/open-ce/open-ce/releases Open-CE Releases]. Currently only python 3.7 and 3.8 are supported.

*Packages can be installed by setting Open-CE conda channel:
<pre>
conda install -c /scinet/mist/ibm/open-ce/1.2.2 python=3.x PACKAGE
</pre>
*Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== PyTorch ==
=== Installing from IBM Conda Channel ===
The easiest way to install PyTorch on Mist is using IBM's Conda channel. User needs to prepare a conda environment and install PyTorch using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n pytorch_env python=3.7
source activate pytorch_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ pytorch=1.3.1
Or
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/ pytorch=1.5.0
</pre>
'''NEWER VERSIONS FROM OPEN-CE:'''
<pre>
module load anaconda3
conda create -n pytorch_env python=3.8 (or 3.7)
source activate pytorch_env
conda install -c /scinet/mist/ibm/open-ce/1.2.2 pytorch=1.7.1
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>
Add below command into your job script before python command to get deterministic results, see details here: [https://github.com/pytorch/pytorch/issues/39849]
<pre>
export CUBLAS_WORKSPACE_CONFIG=:4096:2
</pre>

== RAPIDS ==
The [https://rapids.ai RAPIDS] is a suite of open source software libraries that gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. The RAPIDS data science framework includes a collection of libraries: '''cuDF(GPU DataFrames)''', '''cuML(GPU Machine Learning Algorithms)''', '''cuStrings(GPU String Manipulation)''', etc.

=== Installing from IBM Conda Channel ===
The easiest way to install RAPIDS on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install powerai-rapids using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n rapids_env python=3.7
source activate rapids_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ powerai-rapids
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== TensorFlow and Keras ==
=== Installing from IBM Conda Channel ===
The easiest way to install TensorFlow and Keras on Mist is using IBM's Conda channel. User needs to prepare a conda environment and install TensorFlow-gpu using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n tf_env python=3.7
source activate tf_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ tensorflow-gpu==2.1.2
Or
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/ tensorflow-gpu==2.2.0
If you need TF 1.x version:
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ tensorflow-gpu==1.15.4
</pre>
'''NEWER VERSIONS FROM OPEN-CE:'''
<pre>
module load anaconda3
conda create -n tf_env python=3.8 (or 3.7)
source activate tf_env
conda install -c /scinet/mist/ibm/open-ce/1.2.2 tensorflow==2.4.2
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

= Testing and debugging =
You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.
* Small test jobs can be run on the login node. Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than one gpu and a few cores.

* Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
mist-login01:~$ debugjob --clean -g G
where G is the number of gpus, If G=1, this gives an interactive session for 2 hours, whereas G=4 gets you a single node with 4 gpus for 30 minutes, and with G=8 (the maximum) gets you 2 nodes each with 4 gpus for 30 minutes. The <tt>--clean</tt> argument is optional but recommended as it will start the session without any modules loaded, thus mimicking more closely what happens when you submit a job script.

= Submitting jobs =
Once you have compiled and tested your code or workflow on the Mist login nodes, and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. Your jobs will run on some of Mist's 53 compute nodes. When and where your job runs is determined by the scheduler.

Mist uses SLURM as its job scheduler. It is configured to allow only '''Single-GPU jobs''' and '''Full-node jobs (4 GPUs per node)'''.

You submit jobs from a login node by passing a script to the sbatch command:

mist-login01:scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course. In most cases, you should not submit from your $HOME directory, but rather, from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

Example job scripts can be found below.
Keep in mind:
* Scheduling is by single gpu or by full node, so you ask only 1 gpu or 4 gpus per node.
* Your job's maximum walltime is 24 hours.
* Jobs must write their output to your scratch or project directory (home is read-only on compute nodes).
* Compute nodes have no internet access.
* Your job script will not remember the modules you have loaded, so it needs to contain "module load" commands of all the required modules (see examples below).
== SOSCIP Users ==
*[https://www.soscip.org SOSCIP] is a consortium to bring together industrial partners and academic researchers and provide them with sophisticated advanced computing technologies and expertise to solve social, technical and business challenges across sectors and drive economic growth.

If you are working on a SOSCIP project, please contact [mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca] to have your user account added to SOSCIP project accounts. SOSCIP users need to submit jobs with additional SLURM flag to get higher priority:
<pre>
#SBATCH -A soscip-<SOSCIP_PROJECT_ID> #e.g. soscip-3-001
OR
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID>
</pre>

== Single-GPU job script ==
For a single GPU job, each will have a quarter of the node which is 1 GPU + 8/32 CPU Cores/Threads + ~58GB CPU memory. '''Users should never ask CPU or Memory explicitly.''' If running MPI program, user can set --ntasks to be the number of MPI ranks. '''Do NOT set --ntasks for non-MPI programs.'''
*It is suggested to use NVIDIA Multi-Process Service (MPS) if running multiple MPI ranks on one GPU.
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate conda_env
python code.py ...
</pre>

== Full-node job script ==
'''If you are not sure the program can be executed on multiple GPUs, please follow the single-gpu job instruction above or contact SciNet/SOSCIP support.'''

Multi-GPU job should ask for a minimum of one full node (4 GPUs). User need to specify "compute_full_node" partition in order to get all resource on a node.
*An example for a 1-node job:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4 #this only affects MPI job
#SBATCH --time=1:00:00
#SBATCH -p compute_full_node
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load <modules you need>
Run your program
</pre>

= Jupyter Notebooks =
SciNet’s [[Jupyter Hub]] is a Niagara-type node; it has a different CPU architecture and no GPUs. Conda environments prepared on Mist will not work there properly. Users who need to use Jupyter Notebook to develop and test some aspects of their workflow can create their own server on the Mist login node and use an SSH tunnel to connect to it from outside. Users who choose to do so have to keep in mind that the login node is a shared resource, and heavy calculations should be done only on compute nodes. Processes (including iPython kernels used by the notebooks) are limited to one hour of total CPU time: idle time will not be counted toward this one hour, and use of multiple cores will count proportionally to the number of cores (i.e. a kernel using all 128 virtual cores on the node will be killed after 28 seconds). Idle notebooks can still burden the node by hogging system and GPU memory, please be mindful of other users and terminate notebooks when work is done.

As an example, let us create a new Conda environment and activate it:
<pre>
module load anaconda3
conda create -n jupyter_env python=3.7
source activate jupyter_env
</pre>
Install the Jupyter Notebook server:
<pre>
conda install notebook
</pre>

== Running the notebook server ==
When the Conda environment is active, enter:
<pre>
jupyter-notebook
</pre>
By default, the Jupyter Notebook server uses port 8888 (can be overridden with the <code>--port</code> option). If another user has already started their own server, the default port may be busy, in which case the server will be listening on a different port. Once launched, the server will output some information to the terminal that will include the actual port number used and a 48-character token. For example:
<pre>http://localhost:8890/?token=54c4090d……</pre>
In this example, the server is listening on port 8890.

== Creating a tunnel ==
In order to access this port remotely (i.e. from your office or home), an [https://en.wikipedia.org/wiki/Tunneling_protocol#Secure_Shell_tunneling SSH tunnel] has to be established. Please refer to your SSH client’s documentation for instructions on how to do that. For the OpenSSH client (standard in most Linux distributions and macOS), a tunnel can be opened in a separate terminal session to the one where the Jupyter Notebook server is running. In the new terminal, issue this command:
<pre>
ssh -L8888:localhost:8890 <username>@mist.scinet.utoronto.ca
</pre>
(replace <code><username></code> with your actual username) The tunnel is open as long as this SSH connection is alive. In this example, we tunnel Mist login node’s port 8890 (where our server is assumed to be running) to our home computer’s port 8888 (any other free port is fine). The notebook can be accessed in the browser at the <code><nowiki>http://localhost:8888</nowiki></code> address (followed by <code>/?token=54c4090d……</code>, or the token can be input on the webpage).

== Using Jupyter on compute nodes ==

You can use the instructions here to set up a Jupyter Notebook server on a compute node (including a [[#Testing_and_debugging|debugjob]]). '''We strongly discourage''' you from running an interactive notebook on a compute node (other than for a debugjob), scheduled jobs run in arbitrary times and are not meant to be interactive. Jupyter notebooks can be run non-interactively or converted to Python scripts.

To launch the Jupyter Notebook server, load the <code>anaconda3</code> module and activate your environment as before (by adding the appropriate lines to the submission script, if you are not using the compute node with an interactive shell). Launching the server has to be done like so:
<pre>
HOME=/dev/shm/$USER jupyter-notebook
</pre>
That is because Jupyter will fail unless it can write to the home folder, which is read-only from compute nodes. This modification of the <code>$HOME</code> environment variable will carry over into the notebooks, which is usually not a problem, but in case the notebook relies on this environment variable (e.g. to read certain files), it can be reset manually in the notebook (<code>import os; os.environ['HOME']=……</code>).

Because compute nodes are not accessible from the Internet, tunneling has to be done twice, once from the remote location (office or home) to the Mist login node, and then from the login node to the compute node. Assuming the server is running on port 8890 of the mist006 node, open the first tunnel in a new terminal session in the remote computer:
<pre>
ssh -L8888:localhost:9999 <username>@mist.scinet.utoronto.ca
</pre>
where 9999 is any available port on the Mist login node (to test port availability enter <code>ss -Hln src :9999</code> in the terminal when connected to the Mist login node; an empty output indicates that the port is free). In the same session in the login node that was created with the above command, open the second tunnel to the compute node:
<pre>
ssh -L9999:localhost:8890 mist006
</pre>
Be aware that the second tunnel will automatically disconnect once the job on the compute node times out or is relinquished. The Jupyter Notebook server running on the compute node can now be accessed from the browser as in the previous subsection.

= Support =

SciNet inquiries:
* [mailto:support@scinet.utoronto.ca support@scinet.utoronto.ca]

SOSCIP inquiries:
*[mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca]

Main Page

2021-07-20T23:09:37Z

Ymeiron:

Mist

2021-07-12T19:48:32Z

Ymeiron: Reverted edits by Ymeiron (talk) to last revision by Feimao

{{Infobox Computer
|image=[[Image:Mist.jpg|center|300px|thumb]]
|name=Mist
|installed=Dec 2019
|operatingsystem= Red Hat Enterprise Linux 7.6
|loginnode= mist.scinet.utoronto.ca
|nnodes= 54 IBM AC922
|rampernode= 256 GB
|gpuspernode=4 V100-SMX2-32GB
|interconnect=Mellanox EDR
|vendorcompilers= NVCC, IBM XL
|queuetype=Slurm
}}

=Specifications=
Mist is a SciNet-[[#SOSCIP Users |SOSCIP]] joint GPU cluster consisting of 54 IBM AC922 servers. Each node of the cluster has 32 IBM Power9 cores, 256GB RAM and 4 NVIDIA V100-SMX2-32GB GPU with NVLINKs in between. The cluster has InfiniBand EDR interconnection providing GPU-Direct RMDA capability.

= Getting started on Mist =
Mist can be accessed directly.
<pre>
ssh -Y MYCCUSERNAME@mist.scinet.utoronto.ca
</pre>
Mist login node '''mist-login01''' can also be accessed via Niagara cluster.
<pre>
ssh -Y MYCCUSERNAME@niagara.scinet.utoronto.ca
ssh -Y mist-login01
</pre>
== Storage ==
The filesystem for Mist is shared with Niagara cluster. See [https://docs.scinet.utoronto.ca/index.php/Niagara_Quickstart#Your_various_directories Niagara Storage] for more details.

= Loading software modules =

You have two options for running code on Mist: use existing software, or compile your own. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]].

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.
== Tips for loading software ==

* We advise '''''against''''' loading modules in your .bashrc. This can lead to very confusing behaviour under certain circumstances. Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>cuda</code> → <code>cuda/11.0.3</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Modules often require other modules to be loaded first. Solve these dependencies by using [[Using_modules#Module_spider | <code>module spider</code>]].

= Available compilers and interpreters =
* <tt>cuda</tt> module has to be loaded first for GPU softwares.
* For most compiled software, one should use the GNU compilers (<tt>gcc</tt> for C, <tt>g++</tt> for C++, and <tt>gfortran</tt> for Fortran). Loading <tt>gcc</tt> module makes these available.
* The IBM XL compiler suite (<tt>xlc_r, xlc++_r, xlf_r</tt>) is also available, if you load one of the <tt>xl</tt> modules.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> or <tt>spectrum-mpi</tt> module.

=== CUDA ===

The current installed CUDA Tookits are '''11.0.3''' and '''10.2.2 (10.2.89)'''
<pre>
module load cuda/11.0.3
module load cuda/10.2.2
</pre>
*A compiler (GCC, XL or NVHPC/PGI) module must be loaded in order to use CUDA to build any code.
The current NVIDIA driver version is 450.119.04.

===GNU Compilers ===

Available GCC modules are:
<pre>
gcc/9.3.0 (must load CUDA 11)
gcc/8.5.0 (must load CUDA 10)
gcc/10.3.0 (w/o CUDA)
</pre>

=== IBM XL Compilers ===

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xl/16.1.1.10
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER9 CPU. Information about the IBM XL Compilers can be found at the following links:[https://www.ibm.com/support/knowledgecenter/SSXVZZ_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++],
[https://www.ibm.com/support/knowledgecenter/SSAT4T_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

=== OpenMPI ===
<tt>openmpi/<version></tt> module is avaiable with different compilers including GCC and XL. <tt>spectrum-mpi/<version></tt> module provides IBM Spectrum MPI.

=== NVHPC/PGI ===
PGI compiler is provided in NVHPC (NVIDIA HPC SDK).
<pre>
module load nvhpc/21.3
</pre>

= Softwares =
== Amber20 ==

Users who hold Amber20 license can build Amber20 from its source code and run on Mist. '''SOSCIP/SciNet doesn't provide Amber license or source code.'''

=== Building Amber20 ===
Modules that are needed for building Amber20:
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 anaconda3/2021.05 cmake/3.19.8
</pre>
Cmake configuration:
<pre>
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/where-amber-install -DCOMPILER=GNU -DMPI=FALSE -DCUDA=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=FALSE -DOPENMP=TRUE -DNCCL=FALSE -DAPPLY_UPDATES=TRUE
</pre>

=== Running Amber20 ===
'''NVIDIA Pascal P100 and later GPUs like V100 do not scale beyond a single GPU'''. It is highly suggest to run Amber20 as a single-gpu job.
A job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP-project-ID>

module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 anaconda3/2021.05
export PATH=$HOME/where-amber-install/bin:$PATH
export LD_LIBRARY_PATH=$HOME/where-amber-install/lib:$LD_LIBRARY_PATH
pmemd.cuda .... <parameters> ...
</pre>

== Anaconda (Python) ==
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation. Anaconda is provided as modules: '''anaconda3'''

To install Anaconda locally, user need to load the module and create a conda environment:
<pre>
module load anaconda3
conda create -n myPythonEnv python=3.8
</pre>
*Note: By default, conda environments are located in '''$HOME/.conda/envs'''. Cache (downloaded tarballs and packages) is under '''$HOME/.conda/pkgs'''. User may run into problem with disk quota if there are too many environments created. To clean conda cache, '''please run: "conda clean -y --all" and "rm -rf $HOME/.conda/pkgs/*" after installation of packages'''.

To activate the conda environment: (should be activated before running python)
<pre>
source activate myPythonEnv
</pre>
Note that you SHOULD NOT use '''conda activate myPythonEnv''' to activate the environment. This leads to all sorts of problems. Once the environment is activated, user can update or install packages via '''conda''' or '''pip'''
<pre>
conda install <package_name> (preferred way to install packages)
pip install <package_name>
</pre>
*Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>
To deactivate:
<pre>
source deactivate
</pre>
To remove a conda environment:
<pre>
conda remove --name myPythonEnv --all
</pre>
To verify that the environment was removed, run:
<pre>
conda info --envs
</pre>

=== Submitting Python Job ===
A single-gpu job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate myPythonEnv
python code.py ...
</pre>

== CuPy ==
[https://cupy.chainer.org CuPy] is an open-source matrix library accelerated with NVIDIA CUDA. It also uses CUDA-related libraries including cuBLAS, cuDNN, cuRand, cuSolver, cuSPARSE, cuFFT and NCCL to make full use of the GPU architecture. CuPy is an implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it. It supports a subset of numpy.ndarray interface.

CuPy can be install into any conda environment. Python packages: numpy, six and fastrlock are required. cuDNN and NCCL are optional.
<pre>
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 nccl/2.9.9 anaconda3/2021.05
conda create -n cupy-env python=3.8 numpy six fastrlock
source activate cupy-env
CFLAGS="-I$MODULE_CUDNN_PREFIX/include -I$MODULE_NCCL_PREFIX/include -I$MODULE_CUDA_PREFIX/include" LDFLAGS="-L$MODULE_CUDNN_PREFIX/lib64 -L$MODULE_NCCL_PREFIX/lib" CUDA_PATH=$MODULE_CUDA_PREFIX pip install cupy
#building/installing CuPy will take a few minutes
</pre>

== Gromacs ==
[http://www.gromacs.org/ GROMACS] is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2019.6
module load MistEnv/2020a cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.6 (old RHEL 7 version for testing only)
</pre>
*'''GROMACS 2020 and 2021''' Thread-MPI version supports full GPU enablement of all key computational sections. The GPU is used throughout the timestep and repeated CPU-GPU transfers are eliminated. Users are suggested to carefully verify the results.
<pre>
module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2020.6
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.2
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 openmpi/4.1.1+ucx-1.10.0 gromacs/2021.2
</pre>
=== Small/Medium Simulation ===
Due to the lack of PME domain decomposition support on GPU, Gromacs uses CPU to calculate PME when using multiple GPUs. '''It is always recommended to use a single GPU to do small and medium sized simulations with Gromacs.''' By using only 1 MPI rank (w/ OpenMP threads) on a single GPU, both non-bonded PP and PME are atomically offloaded to GPU when possible.
* Gromacs 2019 example:
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1

module load MistEnv/2021a cuda/10.2.2 gcc/8.5.0 gromacs/2019.6
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
gmx mdrun -pin off -ntmpi 1 -ntomp 8 ... <other parameters>
</pre>

* Gromacs 2020 or 2021 example:
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 gromacs/2021.2
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
gmx mdrun -pin off -ntmpi 1 -ntomp 8 -update gpu ... <other parameters>
</pre>

=== Large Simulation ===
If memory size (~58GB) for single-gpu job is not sufficient for the simulation, multiple GPUs can be used. It is suggested to test starting with one full node with 4GPUs and force PME on GPU. Multiple PME ranks are not supported with PME on GPU, so if GPU is used for the PME calculation -npme (number of PME ranks) must be set to 1. If PME has less work than PP, it is suggested to run multiple ranks per GPU, so the GPU for PME rank can also do some work on PP rank(s). When running multiple MPI ranks on the same GPU, NVIDIA Multi-Process Service (MPS) must be enabled.
*An example using 4 GPUs, 7 PP ranks + 1 PME rank: ('''-pin on -pme gpu -npme 1''' must be added to mdrun command in order to force GPU to do PME)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 openmpi/4.1.1+ucx-1.10.0 gromacs/2021.2

mkdir -p /dev/shm/nvidia-mps
export CUDA_MPS_PIPE_DIRECTORY=/dev/shm/nvidia-mps
mkdir -p /dev/shm/nvidia-log
export CUDA_MPS_LOG_DIRECTORY=/dev/shm/nvidia-log
nvidia-cuda-mps-control -d

export OMP_NUM_THREADS=4
mpirun -bind-to none gmx_mpi mdrun -pin on -pme gpu -npme 1 ... <add your parameters>
</pre>
*It is suggested to also test using '''--ntasks=4''' and '''OMP_NUM_THREADS=8''' if you receive a NOTE in Gromacs output saying "% performance was lost because the PME ranks had more work to do than the PP ranks". In this case, NVIDIA MPS is not needed since there is only one MPI rank per GPU.
*'''Please note that the solving of PME on GPU is still only the initial version supporting this behaviour, and comes with a set of limitations outlined further below.'''
<pre>
* Only a PME order of 4 is supported on GPUs.
* PME will run on a GPU only when exactly one rank has a PME task, ie. decompositions with multiple ranks doing PME are not supported.
* Only single precision is supported.
* Free energy calculations where charges are perturbed are not supported, because only single PME grids can be calculated.
* Only dynamical integrators are supported (ie. leap-frog, Velocity Verlet, stochastic dynamics)
* LJ PME is not supported on GPUs.
</pre>
*An example using 4 GPUs, '''PME on CPU''': ('''-pin on''' must be added to mdrun command for proper CPU thread bindings)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 openmpi/4.1.1+ucx-1.10.0 gromacs/2021.2

mkdir -p /dev/shm/nvidia-mps
export CUDA_MPS_PIPE_DIRECTORY=/dev/shm/nvidia-mps
mkdir -p /dev/shm/nvidia-log
export CUDA_MPS_LOG_DIRECTORY=/dev/shm/nvidia-log
nvidia-cuda-mps-control -d

export OMP_NUM_THREADS=4
mpirun -bind-to none gmx_mpi mdrun -pin on ... <add your parameters>

# "--ntasks=16, OMP_NUM_THREADS=2" and "--ntasks=4, OMP_NUM_THREADS=8" should also be tested.
# num_Tasks(MPI_ranks) * num_OpenMP_threads = 32
</pre>
*'''NOTE: The above examples will NOT work with multiple nodes. If simulation is too large for a single GPU node, please contact SciNet/SOSCIP support.'''

== NAMD ==
[http://www.ks.uiuc.edu/Research/namd/ NAMD] is a parallel, object-oriented molecular dynamics code designed for high-performance simulation of large biomolecular systems.
=== 2.14 ===
<pre>
module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
</pre>
==== Running with single GPU ====
If you have many jobs to run, it is always suggested to run with a single gpu per job. This makes jobs easier to be scheduled and gives better overall performance.
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=1
#SBATCH --nodes=1

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -bind-to none -hostfile nodelist-$SLURM_JOB_ID `which namd2` +idlepoll +ppn 8 +p 8 stmv.namd
</pre>

==== Running with one process per node (4 GPUs)====
An example of the job script (using 1 node, '''one process per node''', 32 CPU threads per process + 4 GPUs per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 32 +p $((32*SLURM_NTASKS)) stmv.namd
</pre>
==== Running with one process per GPU (4 GPUs)====
NAMD may scale better if using '''one process per GPU'''. Please do your own benchmark.
An example of the job script (using 1 node, '''one process per GPU''', 8 CPU threads per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load MistEnv/2021a cuda/11.0.3 gcc/9.4.0 spectrum-mpi/10.4.0 namd/2.14
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 4 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 8 +p $((8*SLURM_NTASKS)) stmv.namd
</pre>

== Open-CE ==
[https://github.com/open-ce/open-ce Open-CE] is an '''IBM''' repo for feedstock collection, environment data, and scripts for building Tensorflow, Pytorch, XGBoost, and other related packages and dependencies. Open-CE is distributed as a '''conda channel''' on Mist cluster.
Available packages and versions are listed here [https://github.com/open-ce/open-ce/releases Open-CE Releases]. Currently only python 3.7 and 3.8 are supported.

*Packages can be installed by setting Open-CE conda channel:
<pre>
conda install -c /scinet/mist/ibm/open-ce/1.2.2 python=3.x PACKAGE
</pre>
*Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== PyTorch ==
=== Installing from IBM Conda Channel ===
The easiest way to install PyTorch on Mist is using IBM's Conda channel. User needs to prepare a conda environment and install PyTorch using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n pytorch_env python=3.7
source activate pytorch_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ pytorch=1.3.1
Or
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/ pytorch=1.5.0
</pre>
'''NEWER VERSIONS FROM OPEN-CE:'''
<pre>
module load anaconda3
conda create -n pytorch_env python=3.8 (or 3.7)
source activate pytorch_env
conda install -c /scinet/mist/ibm/open-ce/1.2.2 pytorch=1.7.1
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>
Add below command into your job script before python command to get deterministic results, see details here: [https://github.com/pytorch/pytorch/issues/39849]
<pre>
export CUBLAS_WORKSPACE_CONFIG=:4096:2
</pre>

== RAPIDS ==
The [https://rapids.ai RAPIDS] is a suite of open source software libraries that gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. The RAPIDS data science framework includes a collection of libraries: '''cuDF(GPU DataFrames)''', '''cuML(GPU Machine Learning Algorithms)''', '''cuStrings(GPU String Manipulation)''', etc.

=== Installing from IBM Conda Channel ===
The easiest way to install RAPIDS on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install powerai-rapids using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n rapids_env python=3.7
source activate rapids_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ powerai-rapids
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== TensorFlow and Keras ==
=== Installing from IBM Conda Channel ===
The easiest way to install TensorFlow and Keras on Mist is using IBM's Conda channel. User needs to prepare a conda environment and install TensorFlow-gpu using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n tf_env python=3.7
source activate tf_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ tensorflow-gpu==2.1.2
Or
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/ tensorflow-gpu==2.2.0
If you need TF 1.x version:
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ tensorflow-gpu==1.15.4
</pre>
'''NEWER VERSIONS FROM OPEN-CE:'''
<pre>
module load anaconda3
conda create -n tf_env python=3.8 (or 3.7)
source activate tf_env
conda install -c /scinet/mist/ibm/open-ce/1.2.2 tensorflow==2.4.2
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

= Testing and debugging =
You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.
* Small test jobs can be run on the login node. Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than one gpu and a few cores.

* Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
mist-login01:~$ debugjob --clean -g G
where G is the number of gpus, If G=1, this gives an interactive session for 2 hours, whereas G=4 gets you a single node with 4 gpus for 30 minutes, and with G=8 (the maximum) gets you 2 nodes each with 4 gpus for 30 minutes. The <tt>--clean</tt> argument is optional but recommended as it will start the session without any modules loaded, thus mimicking more closely what happens when you submit a job script.

= Submitting jobs =
Once you have compiled and tested your code or workflow on the Mist login nodes, and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. Your jobs will run on some of Mist's 53 compute nodes. When and where your job runs is determined by the scheduler.

Mist uses SLURM as its job scheduler. It is configured to allow only '''Single-GPU jobs''' and '''Full-node jobs (4 GPUs per node)'''.

You submit jobs from a login node by passing a script to the sbatch command:

mist-login01:scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course. In most cases, you should not submit from your $HOME directory, but rather, from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

Example job scripts can be found below.
Keep in mind:
* Scheduling is by single gpu or by full node, so you ask only 1 gpu or 4 gpus per node.
* Your job's maximum walltime is 24 hours.
* Jobs must write their output to your scratch or project directory (home is read-only on compute nodes).
* Compute nodes have no internet access.
* Your job script will not remember the modules you have loaded, so it needs to contain "module load" commands of all the required modules (see examples below).
== SOSCIP Users ==
*[https://www.soscip.org SOSCIP] is a consortium to bring together industrial partners and academic researchers and provide them with sophisticated advanced computing technologies and expertise to solve social, technical and business challenges across sectors and drive economic growth.

If you are working on a SOSCIP project, please contact [mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca] to have your user account added to SOSCIP project accounts. SOSCIP users need to submit jobs with additional SLURM flag to get higher priority:
<pre>
#SBATCH -A soscip-<SOSCIP_PROJECT_ID> #e.g. soscip-3-001
OR
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID>
</pre>

== Single-GPU job script ==
For a single GPU job, each will have a quarter of the node which is 1 GPU + 8/32 CPU Cores/Threads + ~58GB CPU memory. '''Users should never ask CPU or Memory explicitly.''' If running MPI program, user can set --ntasks to be the number of MPI ranks. '''Do NOT set --ntasks for non-MPI programs.'''
*It is suggested to use NVIDIA Multi-Process Service (MPS) if running multiple MPI ranks on one GPU.
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate conda_env
python code.py ...
</pre>

== Full-node job script ==
'''If you are not sure the program can be executed on multiple GPUs, please follow the single-gpu job instruction above or contact SciNet/SOSCIP support.'''

Multi-GPU job should ask for a minimum of one full node (4 GPUs). User need to specify "compute_full_node" partition in order to get all resource on a node.
*An example for a 1-node job:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4 #this only affects MPI job
#SBATCH --time=1:00:00
#SBATCH -p compute_full_node
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load <modules you need>
Run your program
</pre>

= Jupyter Notebooks =
SciNet’s [[Jupyter Hub]] is a Niagara-type node; it has a different CPU architecture and no GPUs. Conda environments prepared on Mist will not work there properly. Users who need to use Jupyter Notebook to develop and test some aspects of their workflow can create their own server on the Mist login node and use an SSH tunnel to connect to it from outside. Users who choose to do so have to keep in mind that the login node is a shared resource, and heavy calculations should be done only on compute nodes. Processes (including iPython kernels used by the notebooks) are limited to one hour of total CPU time: idle time will not be counted toward this one hour, and use of multiple cores will count proportionally to the number of cores (i.e. a kernel using all 128 virtual cores on the node will be killed after 28 seconds). Idle notebooks can still burden the node by hogging system and GPU memory, please be mindful of other users and terminate notebooks when work is done.

As an example, let us create a new Conda environment and activate it:
<pre>
module load anaconda3
conda create -n jupyter_env python=3.7
source activate jupyter_env
</pre>
Install the Jupyter Notebook server:
<pre>
conda install notebook
</pre>

== Running the notebook server ==
When the Conda environment is active, enter:
<pre>
jupyter-notebook
</pre>
By default, the Jupyter Notebook server uses port 8888 (can be overridden with the <code>--port</code> option). If another user has already started their own server, the default port may be busy, in which case the server will be listening on a different port. Once launched, the server will output some information to the terminal that will include the actual port number used and a 48-character token. For example:
<pre>http://localhost:8890/?token=54c4090d……</pre>
In this example, the server is listening on port 8890.

== Creating a tunnel ==
In order to access this port remotely (i.e. from your office or home), an [https://en.wikipedia.org/wiki/Tunneling_protocol#Secure_Shell_tunneling SSH tunnel] has to be established. Please refer to your SSH client’s documentation for instructions on how to do that. For the OpenSSH client (standard in most Linux distributions and macOS), a tunnel can be opened in a separate terminal session to the one where the Jupyter Notebook server is running. In the new terminal, issue this command:
<pre>
ssh -L8888:localhost:8890 <username>@mist.scinet.utoronto.ca
</pre>
(replace <code><username></code> with your actual username) The tunnel is open as long as this SSH connection is alive. In this example, we tunnel Mist login node’s port 8890 (where our server is assumed to be running) to our home computer’s port 8888 (any other free port is fine). The notebook can be accessed in the browser at the <code><nowiki>http://localhost:8888</nowiki></code> address (followed by <code>/?token=54c4090d……</code>, or the token can be input on the webpage).

== Using Jupyter on compute nodes ==

You can use the instructions here to set up a Jupyter Notebook server on a compute node (including a [[#Testing_and_debugging|debugjob]]). '''We strongly discourage''' you from running an interactive notebook on a compute node (other than for a debugjob), scheduled jobs run in arbitrary times and are not meant to be interactive. Jupyter notebooks can be run non-interactively or converted to Python scripts.

To launch the Jupyter Notebook server, load the <code>anaconda3</code> module and activate your environment as before (by adding the appropriate lines to the submission script, if you are not using the compute node with an interactive shell). Launching the server has to be done like so:
<pre>
HOME=/dev/shm/$USER jupyter-notebook
</pre>
That is because Jupyter will fail unless it can write to the home folder, which is read-only from compute nodes. This modification of the <code>$HOME</code> environment variable will carry over into the notebooks, which is usually not a problem, but in case the notebook relies on this environment variable (e.g. to read certain files), it can be reset manually in the notebook (<code>import os; os.environ['HOME']=……</code>).

Because compute nodes are not accessible from the Internet, tunneling has to be done twice, once from the remote location (office or home) to the Mist login node, and then from the login node to the compute node. Assuming the server is running on port 8890 of the mist006 node, open the first tunnel in a new terminal session in the remote computer:
<pre>
ssh -L8888:localhost:9999 <username>@mist.scinet.utoronto.ca
</pre>
where 9999 is any available port on the Mist login node (to test port availability enter <code>ss -Hln src :9999</code> in the terminal when connected to the Mist login node; an empty output indicates that the port is free). In the same session in the login node that was created with the above command, open the second tunnel to the compute node:
<pre>
ssh -L9999:localhost:8890 mist006
</pre>
Be aware that the second tunnel will automatically disconnect once the job on the compute node times out or is relinquished. The Jupyter Notebook server running on the compute node can now be accessed from the browser as in the previous subsection.

= Support =

SciNet inquiries:
* [mailto:support@scinet.utoronto.ca support@scinet.utoronto.ca]

SOSCIP inquiries:
*[mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca]

Mist

2021-07-07T17:57:06Z

Ymeiron: Reverted edits by Feimao (talk) to last revision by Ymeiron

{{Infobox Computer
|image=[[Image:Mist.jpg|center|300px|thumb]]
|name=Mist
|installed=Dec 2019
|operatingsystem= Red Hat Enterprise Linux 7.6
|loginnode= mist.scinet.utoronto.ca
|nnodes= 54 IBM AC922
|rampernode= 256 GB
|gpuspernode=4 V100-SMX2-32GB
|interconnect=Mellanox EDR
|vendorcompilers= NVCC, IBM XL
|queuetype=Slurm
}}

=Specifications=
Mist is a SciNet-[[#SOSCIP Users |SOSCIP]] joint GPU cluster consisting of 54 IBM AC922 servers. Each node of the cluster has 32 IBM Power9 cores, 256GB RAM and 4 NVIDIA V100-SMX2-32GB GPU with NVLINKs in between. The cluster has InfiniBand EDR interconnection providing GPU-Direct RMDA capability.

= Getting started on Mist =
Mist can be accessed directly.
<pre>
ssh -Y MYCCUSERNAME@mist.scinet.utoronto.ca
</pre>
Mist login node '''mist-login01''' can also be accessed via Niagara cluster.
<pre>
ssh -Y MYCCUSERNAME@niagara.scinet.utoronto.ca
ssh -Y mist-login01
</pre>
== Storage ==
The filesystem for Mist is shared with Niagara cluster. See [https://docs.scinet.utoronto.ca/index.php/Niagara_Quickstart#Your_various_directories Niagara Storage] for more details.

= Loading software modules =

You have two options for running code on Mist: use existing software, or compile your own. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]].

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.
== Tips for loading software ==

* We advise '''''against''''' loading modules in your .bashrc. This can lead to very confusing behaviour under certain circumstances. Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>cuda</code> → <code>cuda/10.1.243</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Modules often require other modules to be loaded first. Solve these dependencies by using [[Using_modules#Module_spider | <code>module spider</code>]].

= Available compilers and interpreters =
* <tt>cuda</tt> module has to be loaded first for GPU softwares.
* For most compiled software, one should use the GNU compilers (<tt>gcc</tt> for C, <tt>g++</tt> for C++, and <tt>gfortran</tt> for Fortran). Loading <tt>gcc</tt> module makes these available.
* The IBM XL compiler suite (<tt>xlc_r, xlc++_r, xlf_r</tt>) is also available, if you load one of the <tt>xl</tt> modules.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> or <tt>spectrum-mpi</tt> module.

=== CUDA ===

The current installed CUDA Tookits are '''10.1.243''' and '''10.2.89 (default)'''
<pre>
module load cuda/<version>
</pre>
*A compiler (GCC, XL or PGI) module must be loaded in order to use CUDA to build any code.
The current NVIDIA driver version is 440.33.01.

===GNU Compilers ===

Available GCC modules are:
<pre>
gcc/7.5.0
gcc/8.4.0
</pre>

=== IBM XL Compilers ===

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xl/16.1.1.3
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER9 CPU. Information about the IBM XL Compilers can be found at the following links:[https://www.ibm.com/support/knowledgecenter/SSXVZZ_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++],
[https://www.ibm.com/support/knowledgecenter/SSAT4T_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

=== OpenMPI ===
<tt>openmpi/<version></tt> module is avaiable with different compilers including GCC and XL. <tt>spectrum-mpi/<version></tt> module provides IBM Spectrum MPI.

=== PGI ===
To load PGI compiler and its own OpenMPI environment, run:
<pre>
module load pgi/19.10
module load pgi-openmpi/3.1.3
</pre>

= Softwares =
== Amber20 ==

Users who hold Amber20 license can build Amber20 from its source code and run on Mist. '''SOSCIP/SciNet doesn't provide Amber license or source code.'''

=== Building Amber20 ===
Modules that are needed for building Amber20:
<pre>
1) MistEnv/2020a (S) 2) cuda/10.2.89 3) gcc/8.4.0 4) cmake/3.16.3 5) openmpi/4.0.3 6) anaconda3/2019.10 7) nccl/2.5.6
</pre>
Cmake configuration:
<pre>
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/where-amber-install -DCOMPILER=GNU -DMPI=TRUE -DCUDA=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=FALSE -DOPENMP=TRUE -DNCCL=TRUE -DAPPLY_UPDATES=TRUE
</pre>

=== Running Amber20 ===
'''NVIDIA Pascal and later GPUs do not scale beyond a single GPU'''. It is highly suggest to run Amber20 as a single-gpu job.
A job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP-project-ID>

module load cuda/10.2.89 gcc/8.4.0 openmpi/4.0.3 nccl/2.5.6
export PATH=$HOME/where-amber-install/bin:$PATH
export LD_LIBRARY_PATH=$HOME/where-amber-install/lib:$LD_LIBRARY_PATH
pmemd.cuda .... <parameters> ...
</pre>

== Anaconda (Python) ==
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation. Anaconda is provided as modules: '''anaconda3'''

To install Anaconda locally, user need to load the module and create a conda environment:
<pre>
module load anaconda3
conda create -n myPythonEnv python=3.7
</pre>
*Note: By default, conda environments are located in '''$HOME/.conda/envs'''. Cache (downloaded tarballs and packages) is under '''$HOME/.conda/pkgs'''. User may run into problem with disk quota if there are too many environments created. To clean conda cache, '''please run: "conda clean -y --all" and "rm -rf $HOME/.conda/pkgs/*" after installation of packages'''.

To activate the conda environment: (should be activated before running python)
<pre>
source activate myPythonEnv
</pre>
Note that you SHOULD NOT use '''conda activate myPythonEnv''' to activate the environment. This leads to all sorts of problems. Once the environment is activated, user can update or install packages via '''conda''' or '''pip'''
<pre>
conda install <package_name> (preferred way to install packages)
pip install <package_name>
</pre>
To deactivate:
<pre>
source deactivate
</pre>
To remove a conda environment:
<pre>
conda remove --name myPythonEnv --all
</pre>
To verify that the environment was removed, run:
<pre>
conda info --envs
</pre>

=== Submitting Python Job ===
A single-gpu job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate myPythonEnv
python code.py ...
</pre>

== CuPy ==
[https://cupy.chainer.org CuPy] is an open-source matrix library accelerated with NVIDIA CUDA. It also uses CUDA-related libraries including cuBLAS, cuDNN, cuRand, cuSolver, cuSPARSE, cuFFT and NCCL to make full use of the GPU architecture. CuPy is an implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it. It supports a subset of numpy.ndarray interface.

CuPy can be install into any conda environment. Python packages: numpy, six and fastrlock are required. cuDNN and NCCL are optional.
<pre>
module load anaconda3/2019.10 cuda/10.2.89 gcc/7.5.0 cudnn/7.6.5.32 nccl/2.5.6
conda create -n cupy-env python=3.7 numpy six fastrlock
source activate cupy-env
CFLAGS="-I$SCINET_CUDNN_ROOT/include -I$SCINET_NCCL_ROOT/include -I$SCINET_CUDA_ROOT/include" LDFLAGS="-L$SCINET_CUDNN_ROOT/lib64 -L$SCINET_NCCL_ROOT/lib" CUDA_PATH=$SCINET_CUDA_ROOT pip install cupy
#building/installing CuPy will take a few minutes
</pre>

== Gromacs ==
[http://www.gromacs.org/ GROMACS] is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.
<pre>
module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.5
module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.6
</pre>
*'''GROMACS 2020''' Thread-MPI version supports full GPU enablement of all key computational sections. The GPU is used throughout the timestep and repeated CPU-GPU transfers are eliminated. '''Currently only single-GPU is supported on Mist'''. Users are suggested to carefully verify the results.
<pre>
module load cuda/10.2.89 gcc/8.4.0 openmpi/4.0.3 gromacs/2020.4
</pre>
=== Small/Medium Simulation ===
Due to the lack of PME domain decomposition support on GPU, Gromacs uses CPU to calculate PME when using multiple GPUs. '''It is always recommended to use a single GPU to do small and medium sized simulations with Gromacs.''' By using only 1 MPI rank (w/ OpenMP threads) on a single GPU, both non-bonded PP and PME are atomically offloaded to GPU when possible.
* A Single-GPU Gromacs job must ask '''--ntasks=32''' even only 1 MPI rank will be launched by mpirun command. '''OMP_PLACES''' must be set to core to force OpenMP threads on physical CPU cores. '''-bind-to none''' and '''-pin off''' must be set to avoid CPU affiliate conflicts among OpenMP, MPI and Gromacs. '''OMP_NUM_THREADS''' must be set to 8 to get optimal performance.
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=1
#SBATCH --ntasks=32

module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.6
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
mpirun -np 1 -bind-to none gmx_mpi mdrun -pin off -ntomp 8 ... <other parameters>
</pre>
* Groamcs 2020 example: (OpenMPI module should to be loaded, but mpirun should NOT be used)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1

module load cuda/10.2.89 gcc/8.4.0 openmpi/4.0.3 gromacs/2020.4
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
gmx mdrun -pin off -ntmpi 1 -ntomp 8 -update gpu ... <other parameters>
</pre>

=== Large Simulation ===
If memory size (~58GB) for single-gpu job is not sufficient for the simulation, multiple GPUs can be used. It is suggested to test starting with one full node with 4GPUs and force PME on GPU. Multiple PME ranks are not supported with PME on GPU, so if GPU is used for the PME calculation -npme (number of PME ranks) must be set to 1. If PME has less work than PP, it is suggested to run multiple ranks per GPU, so the GPU for PME rank can also do some work on PP rank(s). When running multiple MPI ranks on the same GPU, NVIDIA Multi-Process Service (MPS) must be enabled.
*An example using 4 GPUs, 7 PP ranks + 1 PME rank: ('''-pin on -pme gpu -npme 1''' must be added to mdrun command in order to force GPU to do PME)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.6

mkdir -p /dev/shm/nvidia-mps
export CUDA_MPS_PIPE_DIRECTORY=/dev/shm/nvidia-mps
mkdir -p /dev/shm/nvidia-log
export CUDA_MPS_LOG_DIRECTORY=/dev/shm/nvidia-log
nvidia-cuda-mps-control -d

export OMP_NUM_THREADS=4
mpirun -bind-to none gmx_mpi mdrun -pin on -pme gpu -npme 1 ... <add your parameters>
</pre>
*It is suggested to also test using '''--ntasks=4''' and '''OMP_NUM_THREADS=8''' if you receive a NOTE in Gromacs output saying "% performance was lost because the PME ranks had more work to do than the PP ranks". In this case, NVIDIA MPS is not needed since there is only one MPI rank per GPU.
*'''Please note that the solving of PME on GPU is still only the initial version supporting this behaviour, and comes with a set of limitations outlined further below.'''
<pre>
* Only a PME order of 4 is supported on GPUs.
* PME will run on a GPU only when exactly one rank has a PME task, ie. decompositions with multiple ranks doing PME are not supported.
* Only single precision is supported.
* Free energy calculations where charges are perturbed are not supported, because only single PME grids can be calculated.
* Only dynamical integrators are supported (ie. leap-frog, Velocity Verlet, stochastic dynamics)
* LJ PME is not supported on GPUs.
</pre>
*An example using 4 GPUs, '''PME on CPU''': ('''-pin on''' must be added to mdrun command for proper CPU thread bindings)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.6

mkdir -p /dev/shm/nvidia-mps
export CUDA_MPS_PIPE_DIRECTORY=/dev/shm/nvidia-mps
mkdir -p /dev/shm/nvidia-log
export CUDA_MPS_LOG_DIRECTORY=/dev/shm/nvidia-log
nvidia-cuda-mps-control -d

export OMP_NUM_THREADS=4
mpirun -bind-to none gmx_mpi mdrun -pin on ... <add your parameters>

# "--ntasks=16, OMP_NUM_THREADS=2" and "--ntasks=4, OMP_NUM_THREADS=8" should also be tested.
# num_Tasks(MPI_ranks) * num_OpenMP_threads = 32
</pre>
*'''NOTE: The above examples will NOT work with multiple nodes. If simulation is too large for a single GPU node, please contact SciNet/SOSCIP support.'''

== IBM Watson Machine Learning Community Edition (PowerAI) ==
[https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/ IBM Watson Machine Learning Community Edition (PowerAI)] contains many popular ML packages including TensorFlow, PyTorch, XGBoost and RAPIDS. It is distributed through IBM Conda channel. To install packages from PowerAI, user needs to specify IBM Conda channel when using Anaconda.
<pre>
module load anaconda3

conda create --name wmlce_env -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda <package_name> (e.g. powerai, tensorflow-gpu, keras, pytorch, powerai-rapids, py-xgboost-gpu, etc)

source activate wmlce_env
</pre>
*The WML CE Early Access Conda channel (https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/) makes new versions of frameworks available in advance of formal WML CE releases. Easy upgrade between packages in the main and Early Access channels is not guaranteed. Using a separate conda environment for Early Access packages is recommended.

== NAMD ==
[http://www.ks.uiuc.edu/Research/namd/ NAMD] is a parallel, object-oriented molecular dynamics code designed for high-performance simulation of large biomolecular systems.
=== v2.13 ===
<pre>
module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
</pre>
==== Running with one process per node====
An example of the job script (using 1 node, '''one process per node''', 32 CPU threads per process + 4 GPUs per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 32 +p $((32*SLURM_NTASKS)) stmv.namd
</pre>
==== Running with one process per GPU ====
NAMD may scale better if using '''one process per GPU'''. Please do your own benchmark.
An example of the job script (using 1 node, '''one process per GPU''', 8 CPU threads per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 4 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 8 +p $((8*SLURM_NTASKS)) stmv.namd
</pre>

== PyTorch ==
=== Installing from IBM Conda Channel ===
The easiest way to install PyTorch on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install PyTorch using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n pytorch_env python=3.7
source activate pytorch_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ pytorch=1.3.1
</pre>
'''FOR NEWER VERSIONS:'''
<pre>
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/ pytorch=1.5.0
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== RAPIDS ==
The [https://rapids.ai RAPIDS] is a suite of open source software libraries that gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. The RAPIDS data science framework includes a collection of libraries: '''cuDF(GPU DataFrames)''', '''cuML(GPU Machine Learning Algorithms)''', '''cuStrings(GPU String Manipulation)''', etc.

=== Installing from IBM Conda Channel ===
The easiest way to install RAPIDS on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install powerai-rapids using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n rapids_env python=3.7
source activate rapids_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ powerai-rapids
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== TensorFlow and Keras ==
=== Installing from IBM Conda Channel ===
The easiest way to install TensorFlow and Keras on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install TensorFlow-gpu using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n tf_env python=3.7
source activate tf_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ tensorflow-gpu==2.1.2
If you need TF 1.x version:
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ tensorflow-gpu==1.15.4
</pre>
'''FOR NEWER VERSIONS:'''
<pre>
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/ tensorflow-gpu==2.2.0
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

= Testing and debugging =
You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.
* Small test jobs can be run on the login node. Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than one gpu and a few cores.

* Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
mist-login01:~$ debugjob --clean -g G
where G is the number of gpus, If G=1, this gives an interactive session for 2 hours, whereas G=4 gets you a single node with 4 gpus for 30 minutes, and with G=8 (the maximum) gets you 2 nodes each with 4 gpus for 30 minutes. The <tt>--clean</tt> argument is optional but recommended as it will start the session without any modules loaded, thus mimicking more closely what happens when you submit a job script.

= Submitting jobs =
Once you have compiled and tested your code or workflow on the Mist login nodes, and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. Your jobs will run on some of Mist's 53 compute nodes. When and where your job runs is determined by the scheduler.

Mist uses SLURM as its job scheduler. It is configured to allow only '''Single-GPU jobs''' and '''Full-node jobs (4 GPUs per node)'''.

You submit jobs from a login node by passing a script to the sbatch command:

mist-login01:scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course. In most cases, you should not submit from your $HOME directory, but rather, from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

Example job scripts can be found below.
Keep in mind:
* Scheduling is by single gpu or by full node, so you ask only 1 gpu or 4 gpus per node.
* Your job's maximum walltime is 24 hours.
* Jobs must write their output to your scratch or project directory (home is read-only on compute nodes).
* Compute nodes have no internet access.
* Your job script will not remember the modules you have loaded, so it needs to contain "module load" commands of all the required modules (see examples below).
== SOSCIP Users ==
*[https://www.soscip.org SOSCIP] is a consortium to bring together industrial partners and academic researchers and provide them with sophisticated advanced computing technologies and expertise to solve social, technical and business challenges across sectors and drive economic growth.

If you are working on a SOSCIP project, please contact [mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca] to have your user account added to SOSCIP project accounts. SOSCIP users need to submit jobs with additional SLURM flag to get higher priority:
<pre>
#SBATCH -A soscip-<SOSCIP_PROJECT_ID> #e.g. soscip-3-001
OR
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID>
</pre>

== Single-GPU job script ==
For a single GPU job, each will have a quarter of the node which is 1 GPU + 8/32 CPU Cores/Threads + ~58GB CPU memory. '''Users should never ask CPU or Memory explicitly.''' If running MPI program, user can set --ntasks to be the number of MPI ranks. '''Do NOT set --ntasks for non-MPI programs.'''
*It is suggested to use NVIDIA Multi-Process Service (MPS) if running multiple MPI ranks on one GPU.
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate conda_env
python code.py ...
</pre>

== Full-node job script ==
'''If you are not sure the program can be executed on multiple GPUs, please follow the single-gpu job instruction above or contact SciNet/SOSCIP support.'''

Multi-GPU job should ask for a minimum of one full node (4 GPUs). User need to specify "compute_full_node" partition in order to get all resource on a node.
*An example for a 2-node, 8-rank OpenMPI job: (Each rank binds to 1 GPU and 8 physical CPU cores in this case)
<pre>
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --time=1:00:00
#SBATCH -p compute_full_node
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5

mpirun -bind-to core -map-by slot:PE=8 -report-bindings ./program
</pre>

= Jupyter Notebooks =
SciNet’s [[Jupyter Hub]] is a Niagara-type node; it has a different CPU architecture and no GPUs. Conda environments prepared on Mist will not work there properly. Users who need to use Jupyter Notebook to develop and test some aspects of their workflow can create their own server on the Mist login node and use an SSH tunnel to connect to it from outside. Users who choose to do so have to keep in mind that the login node is a shared resource, and heavy calculations should be done only on compute nodes. Processes (including iPython kernels used by the notebooks) are limited to one hour of total CPU time: idle time will not be counted toward this one hour, and use of multiple cores will count proportionally to the number of cores (i.e. a kernel using all 128 virtual cores on the node will be killed after 28 seconds). Idle notebooks can still burden the node by hogging system and GPU memory, please be mindful of other users and terminate notebooks when work is done.

As an example, let us create a new Conda environment and activate it:
<pre>
module load anaconda3
conda create -n jupyter_env python=3.7
source activate jupyter_env
</pre>
Install the Jupyter Notebook server:
<pre>
conda install notebook
</pre>

== Running the notebook server ==
When the Conda environment is active, enter:
<pre>
jupyter-notebook
</pre>
By default, the Jupyter Notebook server uses port 8888 (can be overridden with the <code>--port</code> option). If another user has already started their own server, the default port may be busy, in which case the server will be listening on a different port. Once launched, the server will output some information to the terminal that will include the actual port number used and a 48-character token. For example:
<pre>http://localhost:8890/?token=54c4090d……</pre>
In this example, the server is listening on port 8890.

== Creating a tunnel ==
In order to access this port remotely (i.e. from your office or home), an [https://en.wikipedia.org/wiki/Tunneling_protocol#Secure_Shell_tunneling SSH tunnel] has to be established. Please refer to your SSH client’s documentation for instructions on how to do that. For the OpenSSH client (standard in most Linux distributions and macOS), a tunnel can be opened in a separate terminal session to the one where the Jupyter Notebook server is running. In the new terminal, issue this command:
<pre>
ssh -L8888:localhost:8890 <username>@mist.scinet.utoronto.ca
</pre>
(replace <code><username></code> with your actual username) The tunnel is open as long as this SSH connection is alive. In this example, we tunnel Mist login node’s port 8890 (where our server is assumed to be running) to our home computer’s port 8888 (any other free port is fine). The notebook can be accessed in the browser at the <code><nowiki>http://localhost:8888</nowiki></code> address (followed by <code>/?token=54c4090d……</code>, or the token can be input on the webpage).

== Using Jupyter on compute nodes ==

You can use the instructions here to set up a Jupyter Notebook server on a compute node (including a [[#Testing_and_debugging|debugjob]]). '''We strongly discourage''' you from running an interactive notebook on a compute node (other than for a debugjob), scheduled jobs run in arbitrary times and are not meant to be interactive. Jupyter notebooks can be run non-interactively or converted to Python scripts.

To launch the Jupyter Notebook server, load the <code>anaconda3</code> module and activate your environment as before (by adding the appropriate lines to the submission script, if you are not using the compute node with an interactive shell). Launching the server has to be done like so:
<pre>
HOME=/dev/shm/$USER jupyter-notebook
</pre>
That is because Jupyter will fail unless it can write to the home folder, which is read-only from compute nodes. This modification of the <code>$HOME</code> environment variable will carry over into the notebooks, which is usually not a problem, but in case the notebook relies on this environment variable (e.g. to read certain files), it can be reset manually in the notebook (<code>import os; os.environ['HOME']=……</code>).

Because compute nodes are not accessible from the Internet, tunneling has to be done twice, once from the remote location (office or home) to the Mist login node, and then from the login node to the compute node. Assuming the server is running on port 8890 of the mist006 node, open the first tunnel in a new terminal session in the remote computer:
<pre>
ssh -L8888:localhost:9999 <username>@mist.scinet.utoronto.ca
</pre>
where 9999 is any available port on the Mist login node (to test port availability enter <code>ss -Hln src :9999</code> in the terminal when connected to the Mist login node; an empty output indicates that the port is free). In the same session in the login node that was created with the above command, open the second tunnel to the compute node:
<pre>
ssh -L9999:localhost:8890 mist006
</pre>
Be aware that the second tunnel will automatically disconnect once the job on the compute node times out or is relinquished. The Jupyter Notebook server running on the compute node can now be accessed from the browser as in the previous subsection.

= Support =

SciNet inquiries:
* [mailto:support@scinet.utoronto.ca support@scinet.utoronto.ca]

SOSCIP inquiries:
*[mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca]

2021-01-25T22:32:56Z

Ymeiron: Changed -A to --account

{{Infobox Computer
|image=[[Image:Mist.jpg|center|300px|thumb]]
|name=Mist
|installed=Dec 2019
|operatingsystem= Red Hat Enterprise Linux 7.6
|loginnode= mist.scinet.utoronto.ca
|nnodes= 54 IBM AC922
|rampernode= 256 GB
|gpuspernode=4 V100-SMX2-32GB
|interconnect=Mellanox EDR
|vendorcompilers= NVCC, IBM XL
|queuetype=Slurm
}}

=Specifications=
Mist is a SciNet-[[#SOSCIP Users |SOSCIP]] joint GPU cluster consisting of 54 IBM AC922 servers. Each node of the cluster has 32 IBM Power9 cores, 256GB RAM and 4 NVIDIA V100-SMX2-32GB GPU with NVLINKs in between. The cluster has InfiniBand EDR interconnection providing GPU-Direct RMDA capability.

= Getting started on Mist =
Mist can be accessed directly.
<pre>
ssh -Y MYCCUSERNAME@mist.scinet.utoronto.ca
</pre>
Mist login node '''mist-login01''' can also be accessed via Niagara cluster.
<pre>
ssh -Y MYCCUSERNAME@niagara.scinet.utoronto.ca
ssh -Y mist-login01
</pre>
== Storage ==
The filesystem for Mist is shared with Niagara cluster. See [https://docs.scinet.utoronto.ca/index.php/Niagara_Quickstart#Your_various_directories Niagara Storage] for more details.

= Loading software modules =

You have two options for running code on Mist: use existing software, or compile your own. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]].

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.
== Tips for loading software ==

* We advise '''''against''''' loading modules in your .bashrc. This can lead to very confusing behaviour under certain circumstances. Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>cuda</code> → <code>cuda/10.1.243</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Modules often require other modules to be loaded first. Solve these dependencies by using [[Using_modules#Module_spider | <code>module spider</code>]].

= Available compilers and interpreters =
* <tt>cuda</tt> module has to be loaded first for GPU softwares.
* For most compiled software, one should use the GNU compilers (<tt>gcc</tt> for C, <tt>g++</tt> for C++, and <tt>gfortran</tt> for Fortran). Loading <tt>gcc</tt> module makes these available.
* The IBM XL compiler suite (<tt>xlc_r, xlc++_r, xlf_r</tt>) is also available, if you load one of the <tt>xl</tt> modules.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> or <tt>spectrum-mpi</tt> module.

=== CUDA ===

The current installed CUDA Tookits are '''10.1.243''' and '''10.2.89 (default)'''
<pre>
module load cuda/<version>
</pre>
*A compiler (GCC, XL or PGI) module must be loaded in order to use CUDA to build any code.
The current NVIDIA driver version is 440.33.01.

===GNU Compilers ===

Available GCC modules are:
<pre>
gcc/7.5.0
gcc/8.4.0
</pre>

=== IBM XL Compilers ===

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xl/16.1.1.3
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER9 CPU. Information about the IBM XL Compilers can be found at the following links:[https://www.ibm.com/support/knowledgecenter/SSXVZZ_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++],
[https://www.ibm.com/support/knowledgecenter/SSAT4T_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

=== OpenMPI ===
<tt>openmpi/<version></tt> module is avaiable with different compilers including GCC and XL. <tt>spectrum-mpi/<version></tt> module provides IBM Spectrum MPI.

=== PGI ===
To load PGI compiler and its own OpenMPI environment, run:
<pre>
module load pgi/19.10
module load pgi-openmpi/3.1.3
</pre>

= Softwares =
== Amber20 ==

Users who hold Amber20 license can build Amber20 from its source code and run on Mist. '''SOSCIP/SciNet doesn't provide Amber license or source code.'''

=== Building Amber20 ===
Modules that are needed for building Amber20:
<pre>
1) MistEnv/2020a (S) 2) cuda/10.2.89 3) gcc/8.4.0 4) cmake/3.16.3 5) openmpi/4.0.3 6) anaconda3/2019.10 7) nccl/2.5.6
</pre>
Cmake configuration:
<pre>
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/where-amber-install -DCOMPILER=GNU -DMPI=TRUE -DCUDA=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=FALSE -DOPENMP=TRUE -DNCCL=TRUE -DAPPLY_UPDATES=TRUE
</pre>

=== Running Amber20 ===
'''NVIDIA Pascal and later GPUs do not scale beyond a single GPU'''. It is highly suggest to run Amber20 as a single-gpu job.
A job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account soscip-<SOSCIP-project-ID>

module load cuda/10.2.89 gcc/8.4.0 openmpi/4.0.3 nccl/2.5.6
export PATH=$HOME/where-amber-install/bin:$PATH
export LD_LIBRARY_PATH=$HOME/where-amber-install/lib:$LD_LIBRARY_PATH
pmemd.cuda .... <parameters> ...
</pre>

== Anaconda (Python) ==
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation. Anaconda is provided as modules: '''anaconda3'''

To install Anaconda locally, user need to load the module and create a conda environment:
<pre>
module load anaconda3
conda create -n myPythonEnv python=3.7
</pre>
*Note: By default, conda environments are located in '''$HOME/.conda/envs'''. Cache (downloaded tarballs and packages) is under '''$HOME/.conda/pkgs'''. User may run into problem with disk quota if there are too many environments created. To clean conda cache, '''please run: "conda clean -y --all" and "rm -rf $HOME/.conda/pkgs/*" after installation of packages'''.

To activate the conda environment: (should be activated before running python)
<pre>
source activate myPythonEnv
</pre>
Note that you SHOULD NOT use '''conda activate myPythonEnv''' to activate the environment. This leads to all sorts of problems. Once the environment is activated, user can update or install packages via '''conda''' or '''pip'''
<pre>
conda install <package_name> (preferred way to install packages)
pip install <package_name>
</pre>
To deactivate:
<pre>
source deactivate
</pre>
To remove a conda environment:
<pre>
conda remove --name myPythonEnv --all
</pre>
To verify that the environment was removed, run:
<pre>
conda info --envs
</pre>

=== Submitting Python Job ===
A single-gpu job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate myPythonEnv
python code.py ...
</pre>

== CuPy ==
[https://cupy.chainer.org CuPy] is an open-source matrix library accelerated with NVIDIA CUDA. It also uses CUDA-related libraries including cuBLAS, cuDNN, cuRand, cuSolver, cuSPARSE, cuFFT and NCCL to make full use of the GPU architecture. CuPy is an implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it. It supports a subset of numpy.ndarray interface.

CuPy can be install into any conda environment. Python packages: numpy, six and fastrlock are required. cuDNN and NCCL are optional.
<pre>
module load anaconda3/2019.10 cuda/10.2.89 gcc/7.5.0 cudnn/7.6.5.32 nccl/2.5.6
conda create -n cupy-env python=3.7 numpy six fastrlock
source activate cupy-env
CFLAGS="-I$SCINET_CUDNN_ROOT/include -I$SCINET_NCCL_ROOT/include -I$SCINET_CUDA_ROOT/include" LDFLAGS="-L$SCINET_CUDNN_ROOT/lib64 -L$SCINET_NCCL_ROOT/lib" CUDA_PATH=$SCINET_CUDA_ROOT pip install cupy
#building/installing CuPy will take a few minutes
</pre>

== Gromacs ==
[http://www.gromacs.org/ GROMACS] is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.
<pre>
module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.5
module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.6
</pre>
*'''GROMACS 2020''' Thread-MPI version supports full GPU enablement of all key computational sections. The GPU is used throughout the timestep and repeated CPU-GPU transfers are eliminated. '''Currently only single-GPU is supported on Mist'''. Users are suggested to carefully verify the results.
<pre>
module load cuda/10.2.89 gcc/8.4.0 openmpi/4.0.3 gromacs/2020.4
</pre>
=== Small/Medium Simulation ===
Due to the lack of PME domain decomposition support on GPU, Gromacs uses CPU to calculate PME when using multiple GPUs. '''It is always recommended to use a single GPU to do small and medium sized simulations with Gromacs.''' By using only 1 MPI rank (w/ OpenMP threads) on a single GPU, both non-bonded PP and PME are atomically offloaded to GPU when possible.
* A Single-GPU Gromacs job must ask '''--ntasks=32''' even only 1 MPI rank will be launched by mpirun command. '''OMP_PLACES''' must be set to core to force OpenMP threads on physical CPU cores. '''-bind-to none''' and '''-pin off''' must be set to avoid CPU affiliate conflicts among OpenMP, MPI and Gromacs. '''OMP_NUM_THREADS''' must be set to 8 to get optimal performance.
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=1
#SBATCH --ntasks=32

module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.6
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
mpirun -np 1 -bind-to none gmx_mpi mdrun -pin off -ntomp 8 ... <other parameters>
</pre>
* Groamcs 2020 example: (OpenMPI module should to be loaded, but mpirun should NOT be used)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1

module load cuda/10.2.89 gcc/8.4.0 openmpi/4.0.3 gromacs/2020.4
export OMP_NUM_THREADS=8
export OMP_PLACES=cores
gmx mdrun -pin off -ntmpi 1 -ntomp 8 -update gpu ... <other parameters>
</pre>

=== Large Simulation ===
If memory size (~58GB) for single-gpu job is not sufficient for the simulation, multiple GPUs can be used. It is suggested to test starting with one full node with 4GPUs and force PME on GPU. Multiple PME ranks are not supported with PME on GPU, so if GPU is used for the PME calculation -npme (number of PME ranks) must be set to 1. If PME has less work than PP, it is suggested to run multiple ranks per GPU, so the GPU for PME rank can also do some work on PP rank(s). When running multiple MPI ranks on the same GPU, NVIDIA Multi-Process Service (MPS) must be enabled.
*An example using 4 GPUs, 7 PP ranks + 1 PME rank: ('''-pin on -pme gpu -npme 1''' must be added to mdrun command in order to force GPU to do PME)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.6

mkdir -p /dev/shm/nvidia-mps
export CUDA_MPS_PIPE_DIRECTORY=/dev/shm/nvidia-mps
mkdir -p /dev/shm/nvidia-log
export CUDA_MPS_LOG_DIRECTORY=/dev/shm/nvidia-log
nvidia-cuda-mps-control -d

export OMP_NUM_THREADS=4
mpirun -bind-to none gmx_mpi mdrun -pin on -pme gpu -npme 1 ... <add your parameters>
</pre>
*It is suggested to also test using '''--ntasks=4''' and '''OMP_NUM_THREADS=8''' if you receive a NOTE in Gromacs output saying "% performance was lost because the PME ranks had more work to do than the PP ranks". In this case, NVIDIA MPS is not needed since there is only one MPI rank per GPU.
*'''Please note that the solving of PME on GPU is still only the initial version supporting this behaviour, and comes with a set of limitations outlined further below.'''
<pre>
* Only a PME order of 4 is supported on GPUs.
* PME will run on a GPU only when exactly one rank has a PME task, ie. decompositions with multiple ranks doing PME are not supported.
* Only single precision is supported.
* Free energy calculations where charges are perturbed are not supported, because only single PME grids can be calculated.
* Only dynamical integrators are supported (ie. leap-frog, Velocity Verlet, stochastic dynamics)
* LJ PME is not supported on GPUs.
</pre>
*An example using 4 GPUs, '''PME on CPU''': ('''-pin on''' must be added to mdrun command for proper CPU thread bindings)
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5 gromacs/2019.6

mkdir -p /dev/shm/nvidia-mps
export CUDA_MPS_PIPE_DIRECTORY=/dev/shm/nvidia-mps
mkdir -p /dev/shm/nvidia-log
export CUDA_MPS_LOG_DIRECTORY=/dev/shm/nvidia-log
nvidia-cuda-mps-control -d

export OMP_NUM_THREADS=4
mpirun -bind-to none gmx_mpi mdrun -pin on ... <add your parameters>

# "--ntasks=16, OMP_NUM_THREADS=2" and "--ntasks=4, OMP_NUM_THREADS=8" should also be tested.
# num_Tasks(MPI_ranks) * num_OpenMP_threads = 32
</pre>
*'''NOTE: The above examples will NOT work with multiple nodes. If simulation is too large for a single GPU node, please contact SciNet/SOSCIP support.'''

== IBM Watson Machine Learning Community Edition (PowerAI) ==
[https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/ IBM Watson Machine Learning Community Edition (PowerAI)] contains many popular ML packages including TensorFlow, PyTorch, XGBoost and RAPIDS. It is distributed through IBM Conda channel. To install packages from PowerAI, user needs to specify IBM Conda channel when using Anaconda.
<pre>
module load anaconda3

conda create --name wmlce_env -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda <package_name> (e.g. powerai, tensorflow-gpu, keras, pytorch, powerai-rapids, py-xgboost-gpu, etc)

source activate wmlce_env
</pre>
*The WML CE Early Access Conda channel (https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/) makes new versions of frameworks available in advance of formal WML CE releases. Easy upgrade between packages in the main and Early Access channels is not guaranteed. Using a separate conda environment for Early Access packages is recommended.

== NAMD ==
[http://www.ks.uiuc.edu/Research/namd/ NAMD] is a parallel, object-oriented molecular dynamics code designed for high-performance simulation of large biomolecular systems.
=== v2.13 ===
<pre>
module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
</pre>
==== Running with one process per node====
An example of the job script (using 1 node, '''one process per node''', 32 CPU threads per process + 4 GPUs per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 32 +p $((32*SLURM_NTASKS)) stmv.namd
</pre>
==== Running with one process per GPU ====
NAMD may scale better if using '''one process per GPU'''. Please do your own benchmark.
An example of the job script (using 1 node, '''one process per GPU''', 8 CPU threads per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 4 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 8 +p $((8*SLURM_NTASKS)) stmv.namd
</pre>

== PyTorch ==
=== Installing from IBM Conda Channel ===
The easiest way to install PyTorch on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install PyTorch using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n pytorch_env python=3.7
source activate pytorch_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ pytorch=1.3.1
</pre>
'''FOR NEWER VERSIONS:'''
<pre>
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/ pytorch=1.5.0
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== RAPIDS ==
The [https://rapids.ai RAPIDS] is a suite of open source software libraries that gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. The RAPIDS data science framework includes a collection of libraries: '''cuDF(GPU DataFrames)''', '''cuML(GPU Machine Learning Algorithms)''', '''cuStrings(GPU String Manipulation)''', etc.

=== Installing from IBM Conda Channel ===
The easiest way to install RAPIDS on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install powerai-rapids using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n rapids_env python=3.7
source activate rapids_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ powerai-rapids
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== TensorFlow and Keras ==
=== Installing from IBM Conda Channel ===
The easiest way to install TensorFlow and Keras on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install TensorFlow-gpu using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n tf_env python=3.7
source activate tf_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ tensorflow-gpu==2.1.2
If you need TF 1.x version:
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ tensorflow-gpu==1.15.4
</pre>
'''FOR NEWER VERSIONS:'''
<pre>
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/ tensorflow-gpu==2.2.0
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

= Testing and debugging =
You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.
* Small test jobs can be run on the login node. Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than one gpu and a few cores.

* Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
mist-login01:~$ debugjob --clean -g G
where G is the number of gpus, If G=1, this gives an interactive session for 2 hours, whereas G=4 gets you a single node with 4 gpus for 30 minutes, and with G=8 (the maximum) gets you 2 nodes each with 4 gpus for 30 minutes. The <tt>--clean</tt> argument is optional but recommended as it will start the session without any modules loaded, thus mimicking more closely what happens when you submit a job script.

= Submitting jobs =
Once you have compiled and tested your code or workflow on the Mist login nodes, and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. Your jobs will run on some of Mist's 53 compute nodes. When and where your job runs is determined by the scheduler.

Mist uses SLURM as its job scheduler. It is configured to allow only '''Single-GPU jobs''' and '''Full-node jobs (4 GPUs per node)'''.

You submit jobs from a login node by passing a script to the sbatch command:

mist-login01:scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course. In most cases, you should not submit from your $HOME directory, but rather, from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

Example job scripts can be found below.
Keep in mind:
* Scheduling is by single gpu or by full node, so you ask only 1 gpu or 4 gpus per node.
* Your job's maximum walltime is 24 hours.
* Jobs must write their output to your scratch or project directory (home is read-only on compute nodes).
* Compute nodes have no internet access.
* Your job script will not remember the modules you have loaded, so it needs to contain "module load" commands of all the required modules (see examples below).
== SOSCIP Users ==
*[https://www.soscip.org SOSCIP] is a consortium to bring together industrial partners and academic researchers and provide them with sophisticated advanced computing technologies and expertise to solve social, technical and business challenges across sectors and drive economic growth.

If you are working on a SOSCIP project, please contact [mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca] to have your user account added to SOSCIP project accounts. SOSCIP users need to submit jobs with additional SLURM flag to get higher priority:
<pre>
#SBATCH -A soscip-<SOSCIP_PROJECT_ID> #e.g. soscip-3-001
OR
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID>
</pre>

== Single-GPU job script ==
For a single GPU job, each will have a quarter of the node which is 1 GPU + 8/32 CPU Cores/Threads + ~58GB CPU memory. '''Users should never ask CPU or Memory explicitly.''' If running MPI program, user can set --ntasks to be the number of MPI ranks. '''Do NOT set --ntasks for non-MPI programs.'''
*It is suggested to use NVIDIA Multi-Process Service (MPS) if running multiple MPI ranks on one GPU.
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate conda_env
python code.py ...
</pre>

== Full-node job script ==
'''If you are not sure the program can be executed on multiple GPUs, please follow the single-gpu job instruction above or contact SciNet/SOSCIP support.'''

Multi-GPU job should ask for a minimum of one full node (4 GPUs). User need to specify "compute_full_node" partition in order to get all resource on a node.
*An example for a 2-node, 8-rank OpenMPI job: (Each rank binds to 1 GPU and 8 physical CPU cores in this case)
<pre>
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --time=1:00:00
#SBATCH -p compute_full_node
#SBATCH --account soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5

mpirun -bind-to core -map-by slot:PE=8 -report-bindings ./program
</pre>

= Support =

SciNet inquiries:
* [mailto:support@scinet.utoronto.ca support@scinet.utoronto.ca]
* [mailto:niagara@computecanada.ca niagara@computecanada.ca]

SOSCIP inquiries:
*[mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca]

Main Page

2020-08-13T12:25:01Z

Ymeiron: