SciNet Users Documentation - User contributions [en]

Ansys

2026-04-29T13:21:43Z

Nolta:

The [http://www.ansys.com/ Ansys] engineering simulation tools are installed in the CC software stack.

=Getting a license=

Licenses are provided by [https://www.cmc.ca CMC Microsystems]. Canadian students and faculty can register at [https://account.cmc.ca/MyAccount/GetAccount.aspx this page].

Once you have an account, you must [https://www.cmc.ca/support-form/ contact CMC] and tell them you want to use the Ansys tools on Trillium, and give them your SciNet username.

Then create the file <code>~/.licenses/ansys.lic</code> containing:

setenv("ANSYSLMD_LICENSE_FILE", "6624@nia-cmc")
setenv("ANSYSLI_SERVERS", "2325@nia-cmc")

You may also use a license server from the other GP clusters (Rorqual, Narval, Fir, Nibi), since Trillium doesn't have its own server. More information can be found here:

https://docs.alliancecan.ca/wiki/Ansys#Configuring_your_license_file

=Running on Trillium=

Load the following modules (current as of Apr 2026, newer versions may be available):

module load StdEnv/2023
module load ansys/2025R2.04

Programs available:

* fluent
* ansysedt
* mapdl
* ...

You can use the Ansys graphical tools to set up your problem, but you cannot use the graphical tools to submit your job. The job must be submitted to the scheduler for running.

More information can be found here: https://docs.computecanada.ca/wiki/ANSYS

==Setting up your .mw directory==

Ansys will attempt to write to your $HOME/.mw directory. This will work when you are testing your workflow on the login nodes, because they can write to $HOME. However, recall that the compute nodes cannot write to the /home filesystem. If you attempt to run Ansys from a compute node using the default configuration, it will fail because Ansys cannot write to $HOME/.mw.

The solution is to create an alternative directory called $SCRATCH/.mw, and create a soft link from $HOME/.mw to $SCRATCH/.mw:
mkdir $SCRATCH/.mw
ln -s $SCRATCH/.mw $HOME/.mw
This will fool Ansys into thinking it is writing to $HOME/.mw, when in fact it is writing to $SCRATCH/.mw. This command only needs to be run once.

==Running ansys==

Example submission script for a job running on 1 node, with max walltime of 11 hours:

<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=192
#SBATCH --time=11:00:00
#SBATCH --job-name test

module load StdEnv/2023
module load ansys/2025R2.04

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

machines=$(srun bash -c 'hostname -s' | sort | uniq | awk '{print $1 ":" 40}' | paste -s -d ':')
ansys252 -b -j JOBNAME -dis -machines "$machines" -i ansys.in
</source>

==Custom license server==

To use your own license server for Ansys, you'll need to open up ssh tunnels to the required ports,
and point Ansys to the tunnels.
For example:

<source lang="bash">
ssh tri-gw -fNL 1055:license.server:1055
ssh tri-gw -fNL 1056:license.server:1056
ssh tri-gw -fNL 2325:license.server:2325

export ANSYSLI_SERVERS=2325@localhost
export ANSYSLMD_LICENSE_FILE=1055@localhost
export LM_LICENSE_FILE=1055@localhost
</source>

Make sure your firewall allows connections from the following IP addresses and confirm with your license provider which ports should be used with the ssh tunnel:

<source>
# 142.150.188.{58..67}
142.150.188.58
142.150.188.59
142.150.188.60
142.150.188.61
142.150.188.62
142.150.188.63
142.150.188.64
142.150.188.65
142.150.188.66
142.150.188.67
</source>

2025-09-17T12:57:23Z

Nolta:

SSH

2024-08-18T13:03:59Z

Nolta: /* Two-Factor authentication */

SSH (secure shell) is the only way to log into the systems at SciNet. It opens a secure, encrypted connection between your computer and SciNet. If you have a Linux or Mac OSX machine, you already have SSH installed; if you have a Windows machine, you will have to install additional software before logging into SciNet.

{{Note|SSH keys are now the only way to authenticate to most SciNet systems, and passwords are not accepted.}}

==SSH For Linux or Mac OS X Users==

===Simple Login===

To login to SciNet's Niagara cluster, open a terminal window and type:

ssh USERNAME@niagara.scinet.utoronto.ca

replacing USERNAME with your SciNet username. Once done, you will be logged into the login nodes at the SciNet data centre, as if you have a terminal from those machines on your desktop.

'''<big>***Note***</big>''' if you chose a custom SSH key name, i.e. something other than the default names: <code>id_dsa</code>, <code>id_ecdsa</code>, <code>id_ed25519</code> and <code>id_rsa</code>, you will need to use the <code>-i</code> option of ssh to specify the path to your private key via:

<pre>
ssh -i /path/to/key USERNAME@niagara.scinet.utoronto.ca
</pre>

More details about custom SSH key names can be found [[#Custom SSH Keys names|here]].

Note that if your username is the same on both the machine you're logging in from and the scinet machines, you can drop the <tt>USERNAME@</tt>, as SSH by default will try to use the username on the machine you are logging in from.

===Copying Files===

The SSH protocol can be used for more than logging in remotely; it can also be used to copy files between machines.

To copy '''small''' files from your home computer to a subdirectory of your <tt>/scratch</tt> directory at SciNet, you would type from a terminal on your computer

scp filetocopy.txt USERNAME@niagara.scinet.utoronto.ca:/scratch/USERNAME/some_subdirectory/

Note that soon the location of your scratch directory will change, and you will have to type:

scp filetocopy.txt USERNAME@niagara.scinet.utoronto.ca:/scratch/G/GROUPNAME/USERNAME/some_subdirectory/

Similarly, to copy files back into your current directory, you would type

scp USERNAME@niagara.scinet.utoronto.ca:/scratch/G/GROUPNAME/USERNAME/my_dirs/myfile.txt .

'''<big>***Note***</big>''' if you chose a custom SSH key name, i.e. something other than the default names: <code>id_dsa</code>, <code>id_ecdsa</code>, <code>id_ed25519</code> and <code>id_rsa</code>, you will need to use the <code>-i</code> option of scp and specify the path to your private key before the file paths via:

<pre>
scp -i /path/to/key filetocopy.txt USERNAME@niagara.scinet.utoronto.ca:/scratch/USERNAME/some_subdirectory/
</pre>

More details about custom SSH key names can be found [[#Custom SSH Keys names|here]].

The [[Niagara_Quickstart#Moving_data | Data Management]] wiki page has much more information on doing large transfers efficiently.

==SSH for Windows Users==

To use SSH on Windows, you will have to install SSH software. SciNet recommends, roughly in order of preference:

* [http://www.cygwin.com/ Cygwin] is an entire linux-like environment for Windows. Using something like Cygwin is highly recommended if you are going to be interacting a lot with linux systems, as it will give you a development environment very similar to that on the systems you'll be using. Download and run <tt>setup.exe</tt>, and install any packages you think you'll need. Once this is done, you will have icons for terminals, including one saying something like "X11". From either of these, you'll be able to type <tt>ssh user@niagara.scinet.utoronto.ca</tt> as above; if you think you will need to pop up windows from SciNet machines (e.g., for displaying data or using [[Performance_And_Debugging_Tools:_GPC | Profiling Tools]]), you'll need to use the X11 terminal and type <tt>ssh -Y user@niagara.scinet.utoronto.ca</tt>. Other ssh tools such as <tt>scp</tt> will work as above.
* [http://mobaxterm.mobatek.net/en/ MobaXterm] is a tabbed ssh client with some Cygwin tools all wrapped up into one executable.
* [http://sshwindows.sourceforge.net/ OpenSSH For Windows] installs only those parts of Cygwin necessary to run SSH. Again, once installed, opening up one of the new terminals allows you to use SSH as in the Linux/Mac OSX section above, but X11 forwarding for displaying windows may not work.
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] is one of the better stand-alone SSH programs for windows. It is a small download, and is enough to get you logged into the SciNet machines. For advanced use like X11 forwarding however, you are better off using Cygwin. A related program, [http://the.earth.li/~sgtatham/putty/latest/x86/pscp.exe PSCP], can be used to copy files using a graphical user interface. '''WARNING:''' Make sure you download putty from the official website, because there are "trojanized" versions of putty around that will send your login information to a site in Russia (as reported [http://blogs.cisco.com/security/trojanized-putty-software here]).

===Copying Files===

To transfer files to Niagara in Windows it is recommended to use the tool called [https://winscp.net/eng/index.php WinSCP]. Setting up a connection to Niagara using your SSH key can be done by following the steps in this [https://www.exavault.com/blog/import-ssh-keys-winscp link].

===Copying Files with scp===
If you want to use <code>scp</code> in Windows (e.g. [http://mobaxterm.mobatek.net/en/ MobaXterm] terminal) and your SSH key was generated in MobaXterm/PuTTY, i.e. your private key has an <code>.ppk</code> extension, then you will need to convert your private key to an OpenSSH format. A good guide on how to do this can be found [https://www.simplified.guide/putty/convert-ppk-to-ssh-key here].

==X11 Forwarding==

If during your login session you will only need to be typing and reading text, the techniques described above will suffice.
However, if in a session you will need to be displaying graphics — such as plotting data on the scinet machines or using our [[Performance_And_Debugging_Tools:_Niagara | performance profiling tools]] — you can use SSH's very powerful ability to forward several different types of data over one connection.
To enable "X11 forwarding" over this SSH connection, add the option <tt>-Y</tt> to your command,

ssh -Y USERNAME@niagara.scinet.utoronto.ca

* Both, Windows and Mac OS users, will need to install an additional program to have X-forwarding working, usually referred to as "Xserver" which will interprete the data (graphics) forwarded and displayed on the local computer.
* Mac OS users need to install [https://www.xquartz.org XQUARTZ]
* Windows users could opt for installing [https://mobaxterm.mobatek.net MobaXterm] which is a ssh-client which already includes an Xserver.

==SSH Keys==

[[SSH | SSH]] has an alternative to passwords to authenticate your login; you can generate a key file on a trusted machine and tell a remote machine to trust logins from a machine that presents that key. This can be both convenient and secure, and may be necessary for some tasks (such as connecting directly to compute nodes to use [[Visualization | some visualization packages]]). Here we describe how to setup keys for logging into SciNet.

In addition to using passwords to [http://en.wikipedia.org/wiki/Authentication authenticate] users, one can use cryptographically secure keys to guarantee that a login request is coming from a trusted account on a remote machine, and automatically allow such requests. Done properly, this is as secure as requiring a password, but can be more convenient, and is necessary for some operations.

===How SSH keys work===

SSH relies on [http://en.wikipedia.org/wiki/Public-key_cryptography public key cryptography] for its encryption. These cryptosystems have a private key, which must be kept secret, and a public key, which may be disseminated freely. In these systems, anyone may use the public key to encode a message; but only the owner of the private key can decode the message. This can also be used to verify identities; if someone is claiming to be Alice, the owner of some private key, Bob can send Alice a message encoded with Alice's well-known public key. If the person claiming to be Alice can then tell Bob what the message really was, then that person at the very least has access to Alice's private key.

To use keys for authentication, you need to:

* Generate a key pair (private and public)
* Copy the public key to a remote site, and add it to the list of authorized keys
* Ensure permissions are set properly
* Test to make sure it works

===Generating an SSH key pair===

The first stage is to create an SSH key pair. On Linux & MacOS (and Windows, with [https://mobaxterm.mobatek.net/ MobaXterm] terminal) this is done using the <tt>ssh-keygen</tt> command:

ssh-keygen -t ed25519

If that doesn't work, try:

ssh-keygen -t rsa -b 4096

This will prompt you for two pieces of information: where to save the key, and a passphrase for the key. The passphrase is like a password, but rather than letting you in to some particular account, it allows you to use the key you've generated to log into other systems.

The default location to save the private key is in <tt>${HOME}/.ssh/id_type</tt> (where <tt>type</tt> is <tt> ed25519</tt> for an Ed25519 key or <tt>rsa</tt> for an RSA key); unless you have some specific reason for placing it elsewhere, use this option. The public key will be <tt>id_type.pub</tt> in the same directory.

While the default names for the keys are sufficient for most use cases, it is worth noticing that you are free to name them as you wish. For example, you might want to include a string referring to your name and where you plan on using this key like <tt>js_cc_ed25519</tt> (and <tt>js_cc_ed25519.pub</tt>), if your name is John Smith and you created a key of <tt>ed25519</tt> type on your laptop to be used on {{Alliance}} HPC systems. This is useful to distinguish amongst several keys you might need to have on the same computer (for example different keys on the same computer to access different clusters).

Your passphrase can be any string, and of any length. It is best not to make it the same as any of your passwords. A reasonably strong passphrase usually constitutes of one to three sentences with a few words each, written with the proper punctuation in place. Make sure it is unique and memorable to you. Do not use any popular catch phrases, jingles, or song choruses. They could be easily found on the web and catalogued on a database for a brute-force attack.

A sample session of generating a key would go like this:

$ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/USERNAME/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/USERNAME/.ssh/id_ed25519.
Your public key has been saved in /home/USERNAME/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:EajOndriRmLpl1qKg03FDhnc0EzRaApdBTygEbpQZrA USERNAME@HOSTNAME
The key's randomart image is:
+--[ED25519 256]--+
|+=*X=*... |
|oB+ O o . |
|E. * o . |
|..+ + . |
|. B . .S |
| = = o |
|.= o.+ |
|o.oo* . |
|..o=.. |
+----[SHA256]-----+

====Don't Use Passphraseless Keys!====

If you do not specify a passphrase, you will have a completely "exposed" private key. '''This is a terrible idea.''' If you then use this key for anything it means that anyone who sits down at your desk, or anyone who borrows or steals your laptop, can login to anywhere you use that key (good guesses could come from just looking at your history) without needing any password, and could do anything they wanted with your account or data. Don't use passphraseless keys.

====Don't Copy Your Private Key to Other Systems!====

A private key should never leave the computer that was used to generate it. If you have a personal computer at home, a laptop and a desktop at work, for example, make sure to repeat the process for generating a public-private key pair on each of these systems. Insert a comment into the key so that you know where the key pair was created and how it was named. It is useful when distinguishing keys coming from different devices when inspecting the public keys. For example, issue the following commands:

$ ssh-keygen -t ed25519 -C "js@mylaptop js_cc" -f $HOME/.ssh/js_cc_ed25519
$ ssh-keygen -t ed25519 -C "js@mydesktop js_cc" -f $HOME/.ssh/js_cc_ed25519
$ ssh-keygen -t ed25519 -C "js@workdesktop js_cc" -f $HOME/.ssh/js_cc_ed25519

for keys created on your personal laptop, desktop and work desktop, respectively (Replace the suggested "js" abbreviations above, or any other reference to John Smith name, to yours). See [https://docs.scinet.utoronto.ca/index.php/SSH_keys#Multiple_SSH_keys Multiple SSH keys] session for other examples.

Private keys are very powerful. In the wrong hands, they can be used to impersonate you on every system you use those keys to access to. Copying them from the location where they were generated to other places also increases the number of systems that that key can be used to compromise. In addition, leaving the private key where it was created protects you against [https://en.wikipedia.org/wiki/Man-in-the-middle_attack man-in-the-middle attacks,] which impersonate the systems you want to copy your keys to and steal your private key in the process.

===Uploading the Public Key to CCDB===

Use your {{Alliance}} credentials to visit the following site:

https://ccdb.computecanada.ca/ssh_authorized_keys

and follow the instructions there to upload your public key. By using the 'cat' command as follows:

cat ~/.ssh/id_ed25519.pub

you can inspect, select and copy the plain text output of your public key to your screen into the CCDB web page above.

Keys uploaded to the CCDB are available by all clusters across the {{DigitalResearchAllianceOfCanada}}.

===<tt>~/.ssh</tt> Directory Permissions===

Note that <tt>SSH</tt> is very fussy about file permissions; your <tt>~/.ssh</tt> directory must only be accessible by you, and your various key files must not be writable (or in some cases, readable) by anyone else. Sometimes users accidentally reset file permissions while editing these files, and problems happen. If you look at the <tt>~/.ssh</tt> directory itself, it should not be accessible by anyone else:

$ ls -ld ~/.ssh
drwx------ 2 USERNAME GROUPNAME 7 Aug 9 15:43 /home/.../.ssh

To fix your permissions, use the following command:

chmod -R go= ~/.ssh/

===Testing Your Key===

Now you should be able to login to the remote system (say, SciNet):

$ ssh USERNAME@niagara.scinet.utoronto.ca
Enter passphrase for key '/home/USERNAME/.ssh/id_ed25519':
Last login: Tue Aug 17 11:24:48 2010 from HOSTNAME
[...]
nia-login07-$

If this is indeed the absolute first time you are trying to access Niagara, for example, please make sure you are actually accessing Niagara by double checking Niagara's login node ssh key fingerprint as instructed [https://docs.scinet.utoronto.ca/index.php/SSH_Changes_in_May_2019 here.] This check is important to avoid being victim of [https://en.wikipedia.org/wiki/Man-in-the-middle_attack man-in-the-middle attacks.] As a reference you can check host key fingerprints for other {{Alliance}} systems at this [https://docs.alliancecan.ca/wiki/SSH_host_keys link.]

If you get the message below you may need to logout of your gnome session and log back in since <tt>ssh-agent</tt> needs to be restarted with the new passphrase ssh key.

$ ssh USERNAME@niagara.scinet.utoronto.ca
Agent admitted failure to sign using the key.

===(Optional) Using <tt>ssh-agent</tt> to Remember Your Key===

But now you've just replaced having to type a password for login with having to type a passphrase for your key; what have you gained?

It turns out that there's an automated way to manage ssh "identities", using the <tt>ssh-agent</tt> command, which should automatically be running on newer Linux or macOS machines. You can add keys to this agent for the duration of your login using the <tt>ssh-add</tt> command:

$ ssh-add
Enter passphrase for /home/USERNAME/.ssh/id_ed25519:
Identity added: /home/USERNAME/.ssh/id_ed25519 (/home/USERNAME/.ssh/id_ed25519)

and then logins will not require the passphrase, as <tt>ssh-agent</tt> will provide access to the key.

When you log out of your home computer, the ssh agent will close, and next time you log in, you will have to <tt>ssh-add</tt> your key. You can also set a timeout of (say) an hour by using <tt>ssh-add -t 3600</tt>. This minimizes the number of times you have to type your passphrase, while still maintaining some degree of key security.

You can list the fingerprints of all identities currently represented by the agent with '-l' option: 'ssh-add -l'. To delete all identities from the agent just type 'ssh-add -D'.

=== Custom SSH Keys names ===

If you use a custom name for your SSH key pair, you will need to create or modify the config file under ~/.ssh/config to something similar:

<pre>
Host niagara
HostName niagara.scinet.utoronto.ca
User YOUR_LOGIN
IdentityFile ~/.ssh/ssh_privatekey_name
</pre>

Then authenticate by typing:

<pre>
$ssh niagara
</pre>

You can also use the -i option of ssh to specify the path to your private key via:

<pre>
$ssh -i /path/to/key USERNAME@niagara.scinet.utoronto.ca
</pre>

=== Multiple SSH keys ===

It's recommended to have different ssh keys for each service, specific role, or domain. For example, to have separate keys for niagara and graham, first generate two new keys:

$ ssh-keygen -t ed25519 -f ~/.ssh/id_niagara -C "Key for Niagara"
...
$ ssh-keygen -t ed25519 -f ~/.ssh/id_graham -C "Key for Graham"
...

Make sure to use different file names for each key. Next, modify your <tt>~/.ssh/config</tt> file, adding <tt>IdentityFile</tt> directives:

<pre>
Host niagara
User myusername
HostName niagara.scinet.utoronto.ca
IdentityFile ~/.ssh/id_niagara

Host graham
User myusername
HostName graham.computecanada.ca
IdentityFile ~/.ssh/id_graham
</pre>

Now when you login with the shortcuts

$ ssh niagara

or

$ ssh graham

different keys will be used.

===Examples===

====Copying a file on another cluster to Niagara====

For convenience I will assume you have already pasted your public key on CCDB so you can access all clusters of the {{DigitalResearchAllianceOfCanada}} with that key. Your modified <tt>~/.ssh/config</tt> file (on your linux laptop
for example) could use then the same IdentityFile directive for all clusters. In this example, let's say Graham and Niagara on {{theAlliance}} use the same key, as well as a remote cluster not part of {{theAlliance}}:
<pre>
Host niagara
User myusername
HostName niagara.scinet.utoronto.ca
IdentityFile ~/.ssh/id_ed25519

Host graham
User myusername
HostName graham.computecanada.ca
IdentityFile ~/.ssh/id_ed25519

Host remote_cluster
User myusername
HostName remote_cluster.other_domain.ca
IdentityFile ~/.ssh/id_ed25519
</pre>

Suppose you want to access Niagara and copy a file on Graham there, or a file on remote_cluster. How would you do it? Based on the assumptions above you could proceed as follows:

1) On your laptop, the first step to follow is to load your private key to the ssh-agent daemon:

$ ssh-add /home/USERNAME/.ssh/id_ed25519
Enter passphrase for /home/USERNAME/.ssh/id_ed25519:
Identity added: /home/USERNAME/.ssh/id_ed25519 (/home/USERNAME/.ssh/id_ed25519)

It is very convenient since you do it once per work session only and it is strongly encouraged.

2) Access Niagara and enable forwarding of the authentication agent connection (ssh-agent):

$ ssh -A niagara

3) Use secure copy command to copy the file on Graham to Niagara:

$ scp -p graham.computecanada.ca:/path/to/file.txt .

'-p' option is commonly used to preserve the date stamp and other file attributes as it is copied. Not mandatory though. Use the '-R' if you want to copy a directory recursively. Be aware though that it might be better to create a tarball first if there are lots of small files in the directory. Note that if you
want to use an alias for graham in the scp command as follows:

$ scp -p graham:/path/to/file.txt .

you would have to copy your modified ~/.ssh/config file above from your laptop to Niagara first.

4) Use secure copy command to copy the file on remote_cluster to Niagara:

$ scp -p remote_cluster.other_domain.ca:/path/to/file.txt .

This assumes you have copied your public key from your laptop to the remote_cluster first:

$ ssh-copy-id -i ~/.ssh/id_ed25519.pub remote_cluster

or in the absence of ssh-copy-id command:

$ cat ~/.ssh/id_ed25519.pub | ssh remote_cluster "cat >> ~/.ssh/authorized_keys"

and also assumes remote_cluster is not part of {{theAlliance}}: therefore it doesn't have a similar mechanism to manage user's ssh keys centrally as CCDB does.

5) The '-A' option should be used deliberately for this copying task for the shortest time needed. The rationale is that if Niagara is compromised an attacker would now be able to use your connection to access other systems using your forwarded ssh-agent. Better to exit and return without any option added to ssh:

$ exit
$ ssh niagara

=== Best Practice Summary ===

* Create one key pair for each computer you work on and give it a meaningful name. In addition, a comment can also help identify the device where they were created.
* Protect each of your private keys with a strong passphrase. We recommend fifteen characters or more.
* Do not share your private keys.
* Never copy your private keys to other systems.
* Create one key pair for each different service, role or domain, and name them accordingly.
* Do not create key pairs in shared systems like HPC clusters.

==SSH Tunnels==

A more-obscure technique for setting up SSH communication is the construction of an SSH tunnel. This can be useful if, for example, your code needs to access an external software license server from a Niagara compute node. You can read about setting up SSH tunnels on Niagara [[SSH_Tunneling|here]].

==Multifactor authentication==

Multifactor authentication via Duo or Yubikey is now required for all users.
Please visit the [https://ccdb.alliancecan.ca/multi_factor_authentications CCDB] to enrol,
and more information can be found [https://docs.alliancecan.ca/wiki/Multifactor_authentication here].

Main Page

2024-06-02T20:23:28Z

Nolta:

Ansys

2024-01-29T20:35:05Z

Docker

2021-07-16T15:04:29Z

Nolta: Created page with " Docker is not available on SciNet's clusters. However, you can use [http://singularity.hpcng.org singularity] to run docker images. == Fetching a docker image == <nowiki>$..."

Docker is not available on SciNet's clusters.
However, you can use [http://singularity.hpcng.org singularity] to run docker images.

== Fetching a docker image ==

<nowiki>$ singularity pull docker://alpine:latest</nowiki>

This pulls the '''alpine:latest''' image from Docker Hub and converts it to singularity's image format,
saving it as a file named '''alpine_latest.sif'''.

As this requires an internet connection to work, it can only be done on the login nodes, and not in job scripts.

You can also pull from other docker registries, e.g.:

<nowiki>$ singularity pull docker://quay.io/biocontainers/samtools:1.13--h8c37831_0</nowiki>

This creates an image file named '''samtools_1.13--h8c37831_0.sif'''.

== Running a command inside the image ==

<nowiki>$ singularity exec alpine_latest.sif cat /etc/alpine-release
3.14.0</nowiki>

<nowiki>$ singularity exec samtools_1.13--h8c37831_0.sif samtools --version
samtools 1.13
Using htslib 1.13
...</nowiki>

Main Page

2021-06-05T19:19:58Z

Nolta:

Main Page

2021-05-27T01:28:51Z

Nolta: /* System Status */

Ansys

2021-02-11T18:00:52Z

Nolta:

The [http://www.ansys.com/ Ansys] engineering simulation tools are installed in both the Niagara and CC software stacks.

=Getting a license=
Licenses are provided by [http://www.cmc.ca CMC Microsystems]. Canadian students and faculty can register at [https://www.cmc.ca/en/MyAccount/GetAccount.aspx this page].

Once you have an account, you must contact CMC and tell them you want to use the Ansys tools on Niagara, and give them your SciNet username.

=Running using the Niagara installation=

==Ansys 2020R2==
Commercial modules can only be accessed using the 'module use' command.

module use /scinet/niagara/software/commercial/modules
module load ansys/2020r2

Programs available:

* fluent
* ansysedt
* mapdl
* ...
You can use the Ansys graphical tools to set up your problem, but you cannot use the graphical tools to submit your job. The job must be submitted to the scheduler for running.

==Setting up your .mw directory==

Ansys will attempt to write to your $HOME/.mw directory. This will work when you are testing your workflow on the login nodes, because they can write to $HOME. However, recall that the compute nodes cannot write to the /home filesystem. If you attempt to run Ansys from a compute node using the default configuration, it will fail because Ansys cannot write to $HOME/.mw.

The solution is to create an alternative directory called $SCRATCH/.mw, and create a soft link from $HOME/.mw to $SCRATCH/.mw:
mkdir $SCRATCH/.mw
ln -s $SCRATCH/.mw $HOME/.mw
This will fool Ansys into thinking it is writing to $HOME/.mw, when in fact it is writing to $SCRATCH/.mw. This command only needs to be run once.

==Running ansys202==

Example submission script for a job running on 1 node, with max walltime of 11 hours:

<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=40
#SBATCH --time=11:00:00
#SBATCH --job-name test

module use /scinet/niagara/software/commercial/modules
module load ansys/2020r2

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

machines=$(srun bash -c 'hostname -s' | sort | uniq | awk '{print $1 ":" 40}' | paste -s -d ':')
ansys202 -b -j JOBNAME -dis -machines "$machines" -i ansys.in
</source>

Installing your own Python Modules

2020-12-05T17:26:43Z

Nolta:

There are many optional and conflicting packages for Python that users could potentially want (see e.g. http://pypi.python.org/pypi). Therefore, users need to install these additional packages locally in their home directories. In fact, there is no choice, as users do not have permissions to install packages system-wide.

Python provides a number of ways to install packages, the most common of which are the <tt>pip</tt> and <tt>conda</tt> commands. By default, these commands would install in the same directory as the one in which the python executable lives,
but python provides a number of ways for users to install libraries in their home directories instead.

One way to do this with <tt>pip</tt> using the <tt>--user</tt> option, but you shouldn't. That approach is now mostly superseded by virtual environments, and we do not recommend using the <tt>--user</tt> option as it can interfere with other Python environments.

Virtual environments are a standard in Python to create isolated Python environments. This is useful when certain modules or certain versions of modules are not available in the default python environment.

Virtual environments can be used either with the [[Python#Regular_Python | regular python modules]] or the [[Python#Intel_Python | intelpython/anaconda]] modules.

== Using Virtualenv in Regular Python ==

===Creation===
First load a python module, e.g.

module load NiaEnv/2019b python/3.6

or

module load NiaEnv/2019b python/3.8

Then create a directory for the virtual environments.
One can put a virtual environment anywhere, but this directory structure is recommended:

mkdir ~/.virtualenvs
cd ~/.virtualenvs

Now we create our first virtualenv called <code>myEnv</code> choose any name you like:

virtualenv --system-site-packages ~/.virtualenvs/myenv

The "--system-site-packages" flag will use the system-installed versions of packages rather than installing them anew (the list of these packages can be found on the [[Python]] wiki page). This will result in fewer files created in your virtual environment. After that you can activate that virtual environment:

source ~/.virtualenvs/myenv/bin/activate

As you are in the virtualenv now, you can just type <code>pip install <required module></code> to install any module into your virtual environment.

To go back to the normal python installation simply type

deactivate

===Command line and job usage===

You need to activate the appropriate environment every time you log in, and at the start of all your jobs scripts. However, the installation of packages only needs to be done once. In the NiaEnv/2019b stack, it is *not* necessary to load the python module before activating the environment, while in the NiaEnv/2018a stack, you need to load the python module before activating the environment.

===Usage of your virtual environment by others===

Sharing a virtual environment with another user is easy. As long as the directory containing the virtual environment is readable by that other user (which on Niagara is the default when that user is in the same group as the directory), then they simply have to source the activate file in the bin directory of that environment, e.g.

source /home/g/group/user/.virtualenvs/myenv/bin/activate

===Usage in the Jupyter Hub===

You can use your virtual environment in Niagara's [[Jupyter_Hub]], but there are two additional steps required to get the JupterHub to know about your environment and to make it as one of its possible "kernels" for new notebooks.

After having activated your environment, execute the following two commands

pip install ipykernel
python -m ipykernel install --name NAME --user
venv2jup

The first installs the packages needed to interface with jupyter as a kernel, the latter puts an entry in the <tt>.share/jupyter</tt> directory, in which the jupyterhub looks for possible kernels. The final command corrects some paths and checks if all is setup properly. This procedure works for NiaEnv/2019b, but may fail for NiaEnv/2018a.

For conda environments that were installed in .conda/envs, the jupyter notebook should pick them up automatically.

== Using Virtual Environments in Intelpython/Anaconda ==

===Creation===

One can use the same kind of virtual environments for the intelpython and conda modules as for regular modules. However,
environments are built-in in Anaconda, see [https://conda.io/docs/user-guide/tasks/manage-environments.html]. These "conda environments" are not the same as regular virtual environments, as they can contain general packages, such as compilers. The latter feature means that conda environments are much more flexible, but also that they do not cooperate well with other software modules on Niagara. Therefore, you should always use regular virtual environments and pip on Niagara and not conda, unless you have a good reason not too.

First, you just need to load a conda-like module, e.g.

module load NiaEnv/2019b intelpython3

Then, you create a virtual environment

conda create -n myPythonEnv python=3.6

(conda puts the environment in the directory <tt>$HOME/.conda/envs/myPythonEnv</tt>)

Next, you activate your conda environment:

source activate myPythonEnv

At this point you are in your own environment and can just do the installation of any package that you need, e.g.

pip install myFAVpackage

or
conda install myFAVpackage

To go back to the normal python installation, type

source deactivate

===Command line and job usage===

You need to load the intelpython/anaconda module and activate the appropriate environment every time you log in, and at the start of all your jobs scripts. However, the installation of packages only needs to be done once.

===Usage in the Jupyter Hub===

You can use conda environment in Niagara's [[Jupyter_Hub]]. If they were installed in .conda/envs, the jupyter notebook should pick them up automatically.

==Installing the Scientific Python Suite==

For many scientific codes the packages ''numpy'', ''scipy'', ''matplotlib'', ''pandas'' and ''ipython'' are used. Versions of these are already in the python modules (except for the regular python modules in the NiaEnv/2018a stack).

However, if you need different versions, you could start your virtual environment without <tt>--system-site-packages</tt>. In that case, for regular python modules, please install versions of package with an <tt>intel-</tt> prefix, if they exists, so that you will get the most optimized version of the package.

Main Page

2020-08-15T02:37:24Z

Nolta:

Main Page

2020-06-29T17:03:52Z

Nolta:

User Ramdisk

2020-02-26T14:25:01Z

Nolta:

On the Niagara nodes a `ramdisk' is available. Up to 70 percent of the RAM on the node (i.e. 202GB) may be used as a temporary file system. This is particularly useful for use in the early stages of migrating desktop-computing codes to a High Performance Computing platform such as Niagara, especially those that use a lot of file I/O (Input/Output). Using a lot of I/O is a bottleneck in large scale computing, especially on parallel file systems (such as the GPFS used on Niagara), since the files are synchronized across the whole network.

Ramdisk is much faster than real disk, and is especially beneficial for codes which perform a lot of small I/O work, since the ramdisk does not require network traffic. However, each node sees its own ramdisk and cannot see files on that of other nodes. You can also not see the ramdisks of the compute nodes on the login nodes. To track progress on a ramdisk, you'd have to [[SSH]] into the respective compute node.

= Using Ramdisk =

To use the ramdisk, create and read to or write from files in /dev/shm/ just as one would to $SCRATCH. Only the amount of RAM needed to store the files will be taken up by the temporary file system. Thus if you have 40 serial jobs each requiring 1 GB of RAM, and 2GB is taken up by various OS services, you would still have approximately 140GB available to use as ramdisk on a 202GB node. However, if you were to write 7 GB of data to the ramdisk, this would exceed available memory and your job would crash.

Note that when using the ramdisk:
* At the start of your job, you can copy frequently accessed files to ramdisk (''stage in''). If there are many such files, it is beneficial to put them in a tar file.
* One would periodically copy the output files from ramdisk to /scratch or /project, as well as at the end of the job (''stage out'').
* It is very important to delete your files from ramdisk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill his job.

== A simple example ==

A simple script using the ramdisk for 40 serial jobs in a 4 hour window might look like this:
<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=40
#SBATCH --time=4:00:00
#SBATCH --job-name ramdisk-example

workdir="/dev/shm/$USER/workdir"

mkdir -p $workdir

cp $SLURM_SUBMIT_DIR/* $workdir

cd $workdir

for ((i=1;i<=40;i++)); do
./executable < $i.in > $i.out &
done
wait

tar cf $SLURM_SUBMIT_DIR/out.tar *.out
rm -r $workdir
</source>

Often collections of serial jobs are run on the ramdisk, see the [[Running_Serial_Jobs_on_Niagara | serial run wiki page]] for more details.

==A more complex example==

A more complete script using the ramdisk in a 1 day OpenMP job that saves output periodically, might look like this:
<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=40
#SBATCH --time=24:00:00
#SBATCH --job-name ramdisk-test

#Job parameters:
execname=job # name of the executable
input_tar=input.tar # tar file with input files and executables
output_tar=out.tar # file in which to store output
input_subdir=indir # sub-directory (within input_tar) with input files
output_subdir=outdir # sub-directory to contain of output files
poll_period=60 # how often check for job completion (in seconds)
save_period=120 # how often to save output (in minutes)

#Track how long everything takes.
date

#Copy to ramdisk
echo "Stage-in: copying files to ramdisk directory /dev/shm/$USER"
mkdir -p /dev/shm/$USER/$output_subdir
cd /dev/shm/$USER
cp $SLURM_SUBMIT_DIR/$input_tar .
tar xf $input_tar
rm -rf $input_tar

#Track how long everything takes.
echo -n "Stage-in completed on "
date

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

#Run on ramdisk
echo "Starting job"
./$execname $input_subdir $output_subdir &
# Store the process id in $pid so we may check if it's still running:
pid=$!

#Note:
# 1. The above launching command is appropriate for a multi-threaded (OpenMP) applications.
# 2. Ramdisk MPI jobs are limited to 1 node as /dev/shm is not shared across nodes.
# 3. For serial jobs, you'd want to start 40 jobs at the same time instead, e.g.
# mkdir -p $output_subdir/1
# ./$execname ${input_subdir}/1 ${output_subdir}/1 &
# pid=$!
# mkdir -p $output_subdir/2
# ./$execname ${input_subdir}/2 ${output_subdir}/2 &
# pid=$pid,$!
#
# etc.
#
# mkdir -p $output_subdir/40
# ./$execname ${input_subdir}/40 ${output_subdir}/40 &
# pid=$pid,$!

#Track how long everything takes.
echo -n "Job started on "
date

function save_results {
echo -n "Copying from directory $output_subdir to file $SLURM_SUBMIT_DIR/$output_tar on "
date
tar cf $output_tar $output_subdir/*
cp $output_tar $SLURM_SUBMIT_DIR
echo -n "Copying of output complete on "
date
}

function cleanup_ramdisk {
echo -n "Cleaning up ramdisk directory /dev/shm/$USER on "
date
rm -rf /dev/shm/$USER
echo -n "done at "
date
}

function trap_term {
echo -n "Trapped term (soft kill) signal on "
date
save_results
cleanup_ramdisk
exit
}

function interruptible_sleep {
# waits for a number of seconds
# argument 1 = number of seconds
# note: just doing `sleep $1' would not be interruptible!
for m in `seq $1`; do
sleep 1
done
}

function is_running {
# check if one or more process is running
# argument 1 = a command separated list of PIDs (no spaces)
ps -p $1 -o pid= | wc -l
}

#trap the termination signal, and call the function 'trap_term' when
# that happens, so results may be saved.
trap "trap_term" TERM

#number of pollings per save period (rounded down):
npoll=$(($save_period*60/$poll_period))

#polling and saving loop
running=$(is_running $pid)
while [ $running -gt 0 ]
do
for n in `seq $npoll`
do
interruptible_sleep $poll_period
running=$(is_running $pid)
if [ $running -eq 0 ]; then
break
fi
done
save_results
done

#Done
cleanup_ramdisk

echo -n "Job finished cleanly on "
date

</source>
Notes with this script:
* The script assumes that the tar file <tt>input.tar</tt> contains the executable <tt>job</tt> and the input files in a subdirectory called <tt>indir</tt> (with further subdirectories for the case of 8 serial jobs).
* The executable is supposed to take the locations of the input and output directory as arguments.
* The trap comment makes sure that the results gets saved and the ramdisk gets flushed even when the jobs gets killed before the end of the script is reached. <tt>trap</tt> is a bash script construction that executes the given command when the script is given, in this case, a TERM signal. The TERM signal is given by the scheduler 30 seconds before your time is up.
* You could also [[Using_Signals|trap signals in your C, C++ or FORTRAN codes]].
* All files are kept in a subdirectory of <tt>/dev/shm</tt>. This makes the clean up simpler, and keeps things tidy when doing small test jobs on the development nodes.

Mist

2020-02-12T22:01:43Z

Nolta: /* Available compilers and interpreters */

{{Infobox Computer
|image=[[Image:Mist.jpg|center|300px|thumb]]
|name=Mist
|installed=Dec 2019
|operatingsystem= Red Hat Enterprise Linux 7.6
|loginnode= mist.scinet.utoronto.ca
|nnodes= 54 IBM AC922
|rampernode= 256 GB
|gpuspernode=4 V100-SMX2-32GB
|interconnect=Mellanox EDR
|vendorcompilers= NVCC, IBM XL
|queuetype=Slurm
}}

=Specifications=
Mist is a SciNet-[[#SOSCIP Users |SOSCIP]] joint GPU cluster consisting of 54 IBM AC922 servers. Each node of the cluster has 32 IBM Power9 cores, 256GB RAM and 4 NVIDIA V100-SMX2-32GB GPU with NVLINKs in between. The cluster has InfiniBand EDR interconnection providing GPU-Direct RMDA capability.

= Getting started on Mist =
Mist will be able to be accessed directly in the near future.
<pre>
ssh -Y MYCCUSERNAME@mist.scinet.utoronto.ca
</pre>
But for now, you must access the mist login node '''mist-login01''' via the Niagara cluster.
<pre>
ssh -Y MYCCUSERNAME@niagara.scinet.utoronto.ca
ssh -Y mist-login01
</pre>
== Storage ==
The filesystem for Mist is shared with Niagara cluster. See [https://docs.scinet.utoronto.ca/index.php/Niagara_Quickstart#Your_various_directories Niagara Storage] for more details.

= Loading software modules =

You have two options for running code on Mist: use existing software, or compile your own. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]].

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.
== Tips for loading software ==

* We advise '''''against''''' loading modules in your .bashrc. This can lead to very confusing behaviour under certain circumstances. Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>cuda</code> → <code>cuda/10.1.243</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Modules often require other modules to be loaded first. Solve these dependencies by using [[Using_modules#Module_spider | <code>module spider</code>]].

= Available compilers and interpreters =
* <tt>cuda</tt> module has to be loaded first for GPU softwares.
* For most compiled software, one should use the GNU compilers (<tt>gcc</tt> for C, <tt>g++</tt> for C++, and <tt>gfortran</tt> for Fortran). Loading <tt>gcc</tt> module makes these available.
* The IBM XL compiler suite (<tt>xlc_r, xlc++_r, xlf_r</tt>) is also available, if you load one of the <tt>xl</tt> modules.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> or <tt>spectrum-mpi</tt> module.

=== CUDA ===

The current installed CUDA Tookits are '''10.1.243''' and '''10.2.89 (default)'''
<pre>
module load cuda/<version>
</pre>
*A compiler (GCC, XL or PGI) module must be loaded in order to use CUDA to build any code.
The current NVIDIA driver version is 440.33.01.

===GNU Compilers ===

Available GCC modules are:
<pre>
gcc/7.5.0
gcc/8.3.0
</pre>

=== IBM XL Compilers ===

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xl/16.1.1.3
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER9 CPU. Information about the IBM XL Compilers can be found at the following links:[https://www.ibm.com/support/knowledgecenter/SSXVZZ_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++],
[https://www.ibm.com/support/knowledgecenter/SSAT4T_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

=== OpenMPI ===
<tt>openmpi/<version></tt> module is avaiable with different compilers including GCC and XL. <tt>spectrum-mpi/<version></tt> module provides IBM Spectrum MPI.

=== PGI ===
To load PGI compiler and its own OpenMPI environment, run:
<pre>
module load pgi/19.10
module load pgi-openmpi/3.1.3
</pre>

= Softwares =
== Anaconda (Python) ==
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation. Anaconda is provided as modules: '''anaconda3'''

To install Anaconda locally, user need to load the module and create a conda environment:
<pre>
module load anaconda3
conda create -n myPythonEnv python=3.7
</pre>
*Note: By default, conda environments are located in '''$HOME/.conda/envs'''. Cache (downloaded tarballs and packages) is under '''$HOME/.conda/pkgs'''. User may run into problem with disk quota if there are too many environments created. To clean conda cache, '''please run: "conda clean -y --all" and "rm -rf $HOME/.conda/pkgs/*" after installation of packages'''.

To activate the conda environment: (should be activated before running python)
<pre>
source activate myPythonEnv
</pre>
Note that you SHOULD NOT use '''conda activate myPythonEnv''' to activate the environment. This leads to all sorts of problems. Once the environment is activated, user can update or install packages via '''conda''' or '''pip'''
<pre>
conda install <package_name> (preferred way to install packages)
pip install <package_name>
</pre>
To deactivate:
<pre>
source deactivate
</pre>
To remove a conda enviroment:
<pre>
conda remove --name myPythonEnv --all
</pre>
To verify that the environment was removed, run:
<pre>
conda info --envs
</pre>

=== Submitting Python Job ===
A single-gpu job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH -A <SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate myPythonEnv
python code.py ...
</pre>

== CuPy ==
[https://cupy.chainer.org CuPy] is an open-source matrix library accelerated with NVIDIA CUDA. It also uses CUDA-related libraries including cuBLAS, cuDNN, cuRand, cuSolver, cuSPARSE, cuFFT and NCCL to make full use of the GPU architecture. CuPy is an implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it. It supports a subset of numpy.ndarray interface.

CuPy can be install into any conda environment. Python packages: numpy, six and fastrlock are required. cuDNN and NCCL are optional.
<pre>
module load anaconda3/2019.10 cuda/10.2.89 gcc/7.5.0 cudnn/7.6.5.32 nccl/2.5.6
conda create -n cupy-env python=3.7 numpy six fastrlock
source activate cupy-env
CFLAGS="-I$SCINET_CUDNN_ROOT/include -I$SCINET_NCCL_ROOT/include -I$SCINET_CUDA_ROOT/include" LDFLAGS="-L$SCINET_CUDNN_ROOT/lib64 -L$SCINET_NCCL_ROOT/lib" CUDA_PATH=$SCINET_CUDA_ROOT pip install cupy
#building/installing CuPy will take a few minutes
</pre>

== IBM Watson Machine Learning Community Edition (PowerAI) ==
[https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/ IBM Watson Machine Learning Community Edition (PowerAI)] contains many popular ML packages including TensorFlow, PyTorch, XGBoost and RAPIDS. It is distributed through IBM Conda channel. To install packages from PowerAI, user needs to specify IBM Conda channel when using Anaconda.
<pre>
module load anaconda3

conda create --name wmlce_env -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda <package_name> (e.g. powerai, tensorflow-gpu, keras, pytorch, powerai-rapids, py-xgboost-gpu, etc)

source activate wmlce_env
</pre>
*The WML CE Early Access Conda channel (https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/) makes new versions of frameworks available in advance of formal WML CE releases. Easy upgrade between packages in the main and Early Access channels is not guaranteed. Using a separate conda environment for Early Access packages is recommended.

== NAMD ==
[http://www.ks.uiuc.edu/Research/namd/ NAMD] is a parallel, object-oriented molecular dynamics code designed for high-performance simulation of large biomolecular systems.
=== v2.13 ===
<pre>
module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
</pre>
==== Running with one process per node====
An example of the job script (using 1 node, '''one process per node''', 32 CPU threads per process + 4 GPUs per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 32 +p $((32*SLURM_NTASKS)) stmv.namd
</pre>
==== Running with one process per GPU ====
NAMD may scale better if using '''one process per GPU'''. Please do your own benchmark.
An example of the job script (using 1 node, '''one process per GPU''', 8 CPU threads per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 4 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 8 +p $((8*SLURM_NTASKS)) stmv.namd
</pre>

== PyTorch ==
=== Installing from IBM Conda Channel ===
The easiest way to install PyTorch on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install PyTorch using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n pytorch_env python=3.7
source activate pytorch_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ pytorch
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== RAPIDS ==
The [https://rapids.ai RAPIDS] is a suite of open source software libraries that gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. The RAPIDS data science framework includes a collection of libraries: '''cuDF(GPU DataFrames)''', '''cuML(GPU Machine Learning Algorithms)''', '''cuStrings(GPU String Manipulation)''', etc.

=== Installing from IBM Conda Channel ===
The easiest way to install RAPIDS on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install powerai-rapids using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n rapids_env python=3.7
source activate rapids_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ powerai-rapids
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== TensorFlow and Keras ==
=== Installing from IBM Conda Channel ===
The easiest way to install TensorFlow and Keras on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install TensorFlow-gpu using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n tf_env python=3.7
source activate tf_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ tensorflow-gpu
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

= Testing and debugging =
You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.
* Small test jobs can be run on the login node. Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than one gpu and a few cores.

* Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
mist-login01:~$ debugjob --clean -g G
where G is the number of gpus, If G=1, this gives an interactive session for 2 hours, whereas G=4 gets you a single node with 4 gpus for 30 minutes, and with G=8 (the maximum) gets you 2 nodes each with 4 gpus for 30 minutes. The <tt>--clean</tt> argument is optional but recommended as it will start the session without any modules loaded, thus mimicking more closely what happens when you submit a job script.

= Submitting jobs =
Once you have compiled and tested your code or workflow on the Mist login nodes, and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. Your jobs will run on some of Mist's 53 compute nodes. When and where your job runs is determined by the scheduler.

Mist uses SLURM as its job scheduler. It is configured to allow only '''Single-GPU jobs''' and '''Full-node jobs (4 GPUs per node)'''.

You submit jobs from a login node by passing a script to the sbatch command:

mist-login01:scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course. In most cases, you should not submit from your $HOME directory, but rather, from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

Example job scripts can be found below.
Keep in mind:
* Scheduling is by single gpu or by full node, so you ask only 1 gpu or 4 gpus per node.
* Your job's maximum walltime is 24 hours.
* Jobs must write their output to your scratch or project directory (home is read-only on compute nodes).
* Compute nodes have no internet access.
* Your job script will not remember the modules you have loaded, so it needs to contain "module load" commands of all the required modules (see examples below).
== SOSCIP Users ==
*[https://www.soscip.org SOSCIP] is a consortium to bring together industrial partners and academic researchers and provide them with sophisticated advanced computing technologies and expertise to solve social, technical and business challenges across sectors and drive economic growth.

If you are working on a SOSCIP project, please contact [mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca] to have your user account added to SOSCIP project accounts. SOSCIP users need to submit jobs with additional SLURM flag to get higher priority:
<pre>
#SBATCH -A <SOSCIP_PROJECT_ID>
</pre>

== Single-GPU job script ==
For a single GPU job, each will have a quarter of the node which is 1 GPU + 8/32 CPU Cores/Threads + ~58GB CPU memory. '''Users should never ask CPU or Memory explicitly.''' If running MPI program, user can set --ntasks to be the number of MPI ranks. It is suggested to use NVIDIA Multi-Process Service (MPS) if running multiple MPI ranks on one GPU.
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH -A <SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate conda_env
python code.py ...
</pre>

== Full-node job script ==
Multi-GPU job should ask for a minimum of one full node. User need to specify "compute_full_node" partition in order to get all resource on a node.
*An example for a 2-node, 8-rank OpenMPI job: (Each rank binds to 1 GPU and 8 physical CPU cores in this case)
<pre>
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --time=1:00:00
#SBATCH -p compute_full_node
#SBATCH -A <SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5

mpirun -bind-to core -map-by slot:PE=8 -report-bindings ./program
</pre>

= Support =

SciNet inquiries:
* [mailto:support@scinet.utoronto.ca support@scinet.utoronto.ca]
* [mailto:niagara@computecanada.ca niagara@computecanada.ca]

SOSCIP inquiries:
*[mailto:soscip-support@scinet.utoronto.ca soscip-support@scinet.utoronto.ca]

Mist

2020-02-12T17:24:36Z

Nolta: /* Full-node job script */

{{Infobox Computer
|image=[[Image:Mist.jpg|center|300px|thumb]]
|name=Mist
|installed=Dec 2019
|operatingsystem= Red Hat Enterprise Linux 7.6
|loginnode= mist.scinet.utoronto.ca
|nnodes= 54 IBM AC922
|rampernode= 256 GB
|gpuspernode=4 V100-SMX2-32GB
|interconnect=Mellanox EDR
|vendorcompilers= IBM XL
|queuetype=Slurm
}}

= Warning =

'''Mist is in early users/beta testing phase. All instructions below are temporary and subject to change.'''

=Specifications=
The Mist cluster is a GPU cluster of 54 IBM AC922 servers each with 32 IBM Power9 cores with 4 NVIDIA V100-SMX2-32GB GPU and NVLINKs in between. Each node of the cluster has 256GB RAM. It has InfiniBand EDR interconnection providing GPU-Direct RMDA capability.

= Getting started on Mist =

Currently Mist is under testing phase. Mist login node '''mist-login01''' can be accessed via Niagara cluster.
<pre>
ssh -Y MYCCUSERNAME@niagara.scinet.utoronto.ca
ssh -Y mist-login01
</pre>
== Storage ==
The filesystem for Mist is shared with Niagara cluster. See [https://docs.scinet.utoronto.ca/index.php/Niagara_Quickstart#Your_various_directories Niagara Storage] for more details.

= Loading software modules =

You have two options for running code on Mist: use existing software, or compile your own. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]].

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.
== Tips for loading software ==

* We advise '''''against''''' loading modules in your .bashrc. This can lead to very confusing behaviour under certain circumstances. Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>cuda</code> → <code>cuda/10.1.243</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Modules often require other modules to be loaded first. Solve these dependencies by using [[Using_modules#Module_spider | <code>module spider</code>]].

= Available compilers and interpreters =
* <tt>cuda</tt> module has to be loaded first for GPU softwares.
* For most compiled software, one should use the GNU compilers (<tt>gcc</tt> for C, <tt>g++</tt> for C++, and <tt>gfortran</tt> for Fortran). Loading an <tt>at</tt> ( IBM Advance Toolchain) or <tt>gcc</tt> module makes these available.
* The IBM XL compiler suite (<tt>xlc_r, xlc++_r, xlf_r</tt>) is also available, if you load one of the <tt>xl</tt> modules.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> or <tt>spectrummpi</tt> module.

=== CUDA ===

The current installed CUDA Tookits are '''10.1.243''' and '''10.2.89 (default)'''
<pre>
module load cuda/<version>
</pre>
*A compiler (GCC, XL or PGI) module must be loaded in order to use CUDA to build any code.
The current NVIDIA driver version is 440.33.01.

===GNU Compilers ===

Available GCC modules are:
<pre>
gcc/7.5.0
gcc/8.3.0
</pre>

=== IBM XL Compilers ===

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xl/16.1.1.3
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER9 CPU. Information about the IBM XL Compilers can be found at the following links:[https://www.ibm.com/support/knowledgecenter/SSXVZZ_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++],
[https://www.ibm.com/support/knowledgecenter/SSAT4T_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

=== OpenMPI ===
<tt>openmpi/<version></tt> module is avaiable with different compilers including GCC and XL. <tt>spectrum-mpi/<version></tt> module provides IBM Spectrum MPI.

=== PGI ===
To load PGI compiler and its own OpenMPI environment, run:
<pre>
module load pgi/19.10
module load pgi-openmpi/3.1.3
</pre>

= Softwares =
== Anaconda (Python) ==
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation. Anaconda is provided as modules: '''anaconda3'''

To install Anaconda locally, user need to load the module and create a conda environment:
<pre>
module load anaconda3
conda create -n myPythonEnv python=3.7
</pre>
*Note: By default, conda environments are located in '''$HOME/.conda/envs'''. Cache (downloaded tarballs and packages) is under '''$HOME/.conda/pkgs'''. User may run into problem with disk quota if there are too many environments created. To clean conda cache, '''please run: "conda clean -y --all" and "rm -rf $HOME/.conda/pkgs/*" after installation of packages'''.

To activate the conda environment: '''(should be activated before running python)'''
<pre>
source activate myPythonEnv
</pre>
Note that you SHOULD NOT use '''conda activate myPythonEnv''' to activate the environment. This leads to all sorts of problems. Once the environment is activated, user can update or install packages via '''conda''' or '''pip'''
<pre>
conda install <package_name> (preferred way to install packages)
pip install <package_name>
</pre>
To deactivate:
<pre>
source deactivate
</pre>
To remove a conda enviroment:
<pre>
conda remove --name myPythonEnv --all
</pre>
To verify that the environment was removed, run:
<pre>
conda info --envs
</pre>

=== Submitting Python Job ===
A single-gpu job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH -A <SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate myPythonEnv
python code.py ...
</pre>

== CuPy ==
[https://cupy.chainer.org CuPy] is an open-source matrix library accelerated with NVIDIA CUDA. It also uses CUDA-related libraries including cuBLAS, cuDNN, cuRand, cuSolver, cuSPARSE, cuFFT and NCCL to make full use of the GPU architecture. CuPy is an implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it. It supports a subset of numpy.ndarray interface.

CuPy can be install into any conda environment. Python packages: numpy, six and fastrlock are required. cuDNN and NCCL are optional.
<pre>
module load anaconda3/2019.10 cuda/10.2.89 gcc/7.5.0 cudnn/7.6.5.32 nccl/2.5.6
conda create -n cupy-env python=3.7 numpy six fastrlock
source activate cupy-env
CFLAGS="-I$SCINET_CUDNN_ROOT/include -I$SCINET_NCCL_ROOT/include -I$SCINET_CUDA_ROOT/include" LDFLAGS="-L$SCINET_CUDNN_ROOT/lib64 -L$SCINET_NCCL_ROOT/lib" CUDA_PATH=$SCINET_CUDA_ROOT pip install cupy
#building/installing CuPy will take a few minutes
</pre>

== IBM Watson Machine Learning Community Edition (PowerAI) ==
[https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/ IBM Watson Machine Learning Community Edition (PowerAI)] contains many popular ML packages including TensorFlow, PyTorch, XGBoost and RAPIDS. It is distributed through IBM Conda channel. To install packages from PowerAI, user needs to specify IBM Conda channel when using Anaconda.
<pre>
module load anaconda3

conda create --name wmlce_env -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda <package_name> (e.g. powerai, tensorflow-gpu, keras, pytorch, powerai-rapids, py-xgboost-gpu, etc)

source activate wmlce_env
</pre>
*The WML CE Early Access Conda channel (https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/) makes new versions of frameworks available in advance of formal WML CE releases. Easy upgrade between packages in the main and Early Access channels is not guaranteed. Using a separate conda environment for Early Access packages is recommended.

== NAMD ==
[http://www.ks.uiuc.edu/Research/namd/ NAMD] is a parallel, object-oriented molecular dynamics code designed for high-performance simulation of large biomolecular systems.
=== v2.13 ===
<pre>
module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
</pre>
==== Running with one process per node====
An example of the job script (using 1 node, '''one process per node''', 32 CPU threads per process + 4 GPUs per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 32 +p $((32*SLURM_NTASKS)) stmv.namd
</pre>
==== Running with one process per GPU ====
NAMD may scale better if using '''one process per GPU'''. Please do your own benchmark.
An example of the job script (using 1 node, '''one process per GPU''', 8 CPU threads per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 4 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 8 +p $((8*SLURM_NTASKS)) stmv.namd
</pre>

== PyTorch ==
=== Installing from IBM Conda Channel ===
The easiest way to install PyTorch on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install PyTorch using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n pytorch_env python=3.7
source activate pytorch_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ pytorch
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== RAPIDS ==
The [https://rapids.ai RAPIDS] is a suite of open source software libraries that gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. The RAPIDS data science framework includes a collection of libraries: '''cuDF(GPU DataFrames)''', '''cuML(GPU Machine Learning Algorithms)''', '''cuStrings(GPU String Manipulation)''', etc.

=== Installing from IBM Conda Channel ===
The easiest way to install RAPIDS on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install powerai-rapids using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n rapids_env python=3.7
source activate rapids_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ powerai-rapids
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== TensorFlow and Keras ==
=== Installing from IBM Conda Channel ===
The easiest way to install TensorFlow and Keras on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install TensorFlow-gpu using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n tf_env python=3.7
source activate tf_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ tensorflow-gpu
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

= Testing and debugging =
You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.
* Small test jobs can be run on the login node. Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than one gpu and a few cores.

* Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
mist-login01:~$ debugjob --clean -g G
where G is the number of gpus, If G=1, this gives an interactive session for 2 hours, whereas G=4 gets you a single node with 4 gpus for 30 minutes, and with G=8 (the maximum) gets you 2 nodes each with 4 gpus for 30 minutes. The <tt>--clean</tt> argument is optional but recommended as it will start the session without any modules loaded, thus mimicking more closely what happens when you submit a job script.

= Submitting jobs =
Once you have compiled and tested your code or workflow on the Mist login nodes, and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. Your jobs will run on some of Mist's 53 compute nodes. When and where your job runs is determined by the scheduler.

Mist uses SLURM as its job scheduler. It is configured to allow only '''Single-GPU jobs''' and '''Full-node jobs (4 GPUs per node)'''.

You submit jobs from a login node by passing a script to the sbatch command:

mist-login01:scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course. In most cases, you should not submit from your $HOME directory, but rather, from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

Example job scripts can be found below.
Keep in mind:
* Scheduling is by single gpu or by full node, so you ask only 1 gpu or 4 gpus per node.
* Your job's maximum walltime is 24 hours.
* Jobs must write their output to your scratch or project directory (home is read-only on compute nodes).
* Compute nodes have no internet access.
* Your job script will not remember the modules you have loaded, so it needs to contain "module load" commands of all the required modules (see examples below).
== SOSCIP Users ==
If you are working on a SOSCIP project, please contact soscip-support@scinet.utoronto.ca to have your user account added to SOSCIP project accounts. SOSCIP users need to submit jobs with additional SLURM flag:
<pre>
#SBATCH -A <SOSCIP_PROJECT_ID>
</pre>
== Single-GPU job script ==
For a single GPU job, each will have a quarter of the node which is 1 GPU + 8/32 CPU Cores/Threads + ~58GB CPU memory. '''Users should never ask CPU or Memory explicitly.''' If running MPI program, user can set --ntasks to be the number of MPI ranks. It is suggested to use NVIDIA Multi-Process Service (MPS) if running multiple MPI ranks on one GPU.
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH -A <SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate conda_env
python code.py ...
</pre>

== Full-node job script ==
Multi-GPU job should ask for a minimum of one full node. User need to specify "compute_full_node" partition in order to get all resource on a node.
*An example for a 2-node, 8-rank OpenMPI job: (Each rank binds to 1 GPU and 8 physical CPU cores in this case)
<pre>
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --time=1:00:00
#SBATCH -p compute_full_node
#SBATCH -A <SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load cuda/10.2.89 gcc/8.3.0 openmpi/3.1.5

mpirun -bind-to core -map-by slot:PE=8 -report-bindings ./program
</pre>

Mist

2020-02-12T17:22:54Z

Nolta: /* PGI */

{{Infobox Computer
|image=[[Image:Mist.jpg|center|300px|thumb]]
|name=Mist
|installed=Dec 2019
|operatingsystem= Red Hat Enterprise Linux 7.6
|loginnode= mist.scinet.utoronto.ca
|nnodes= 54 IBM AC922
|rampernode= 256 GB
|gpuspernode=4 V100-SMX2-32GB
|interconnect=Mellanox EDR
|vendorcompilers= IBM XL
|queuetype=Slurm
}}

= Warning =

'''Mist is in early users/beta testing phase. All instructions below are temporary and subject to change.'''

=Specifications=
The Mist cluster is a GPU cluster of 54 IBM AC922 servers each with 32 IBM Power9 cores with 4 NVIDIA V100-SMX2-32GB GPU and NVLINKs in between. Each node of the cluster has 256GB RAM. It has InfiniBand EDR interconnection providing GPU-Direct RMDA capability.

= Getting started on Mist =

Currently Mist is under testing phase. Mist login node '''mist-login01''' can be accessed via Niagara cluster.
<pre>
ssh -Y MYCCUSERNAME@niagara.scinet.utoronto.ca
ssh -Y mist-login01
</pre>
== Storage ==
The filesystem for Mist is shared with Niagara cluster. See [https://docs.scinet.utoronto.ca/index.php/Niagara_Quickstart#Your_various_directories Niagara Storage] for more details.

= Loading software modules =

You have two options for running code on Mist: use existing software, or compile your own. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]].

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.
== Tips for loading software ==

* We advise '''''against''''' loading modules in your .bashrc. This can lead to very confusing behaviour under certain circumstances. Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>cuda</code> → <code>cuda/10.1.243</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Modules often require other modules to be loaded first. Solve these dependencies by using [[Using_modules#Module_spider | <code>module spider</code>]].

= Available compilers and interpreters =
* <tt>cuda</tt> module has to be loaded first for GPU softwares.
* For most compiled software, one should use the GNU compilers (<tt>gcc</tt> for C, <tt>g++</tt> for C++, and <tt>gfortran</tt> for Fortran). Loading an <tt>at</tt> ( IBM Advance Toolchain) or <tt>gcc</tt> module makes these available.
* The IBM XL compiler suite (<tt>xlc_r, xlc++_r, xlf_r</tt>) is also available, if you load one of the <tt>xl</tt> modules.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> or <tt>spectrummpi</tt> module.

=== CUDA ===

The current installed CUDA Tookits are '''10.1.243''' and '''10.2.89 (default)'''
<pre>
module load cuda/<version>
</pre>
*A compiler (GCC, XL or PGI) module must be loaded in order to use CUDA to build any code.
The current NVIDIA driver version is 440.33.01.

===GNU Compilers ===

Available GCC modules are:
<pre>
gcc/7.5.0
gcc/8.3.0
</pre>

=== IBM XL Compilers ===

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xl/16.1.1.3
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER9 CPU. Information about the IBM XL Compilers can be found at the following links:[https://www.ibm.com/support/knowledgecenter/SSXVZZ_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++],
[https://www.ibm.com/support/knowledgecenter/SSAT4T_16.1.1/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

=== OpenMPI ===
<tt>openmpi/<version></tt> module is avaiable with different compilers including GCC and XL. <tt>spectrum-mpi/<version></tt> module provides IBM Spectrum MPI.

=== PGI ===
To load PGI compiler and its own OpenMPI environment, run:
<pre>
module load pgi/19.10
module load pgi-openmpi/3.1.3
</pre>

= Softwares =
== Anaconda (Python) ==
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation. Anaconda is provided as modules: '''anaconda3'''

To install Anaconda locally, user need to load the module and create a conda environment:
<pre>
module load anaconda3
conda create -n myPythonEnv python=3.7
</pre>
*Note: By default, conda environments are located in '''$HOME/.conda/envs'''. Cache (downloaded tarballs and packages) is under '''$HOME/.conda/pkgs'''. User may run into problem with disk quota if there are too many environments created. To clean conda cache, '''please run: "conda clean -y --all" and "rm -rf $HOME/.conda/pkgs/*" after installation of packages'''.

To activate the conda environment: '''(should be activated before running python)'''
<pre>
source activate myPythonEnv
</pre>
Note that you SHOULD NOT use '''conda activate myPythonEnv''' to activate the environment. This leads to all sorts of problems. Once the environment is activated, user can update or install packages via '''conda''' or '''pip'''
<pre>
conda install <package_name> (preferred way to install packages)
pip install <package_name>
</pre>
To deactivate:
<pre>
source deactivate
</pre>
To remove a conda enviroment:
<pre>
conda remove --name myPythonEnv --all
</pre>
To verify that the environment was removed, run:
<pre>
conda info --envs
</pre>

=== Submitting Python Job ===
A single-gpu job example:
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH -A <SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate myPythonEnv
python code.py ...
</pre>

== CuPy ==
[https://cupy.chainer.org CuPy] is an open-source matrix library accelerated with NVIDIA CUDA. It also uses CUDA-related libraries including cuBLAS, cuDNN, cuRand, cuSolver, cuSPARSE, cuFFT and NCCL to make full use of the GPU architecture. CuPy is an implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it. It supports a subset of numpy.ndarray interface.

CuPy can be install into any conda environment. Python packages: numpy, six and fastrlock are required. cuDNN and NCCL are optional.
<pre>
module load anaconda3/2019.10 cuda/10.2.89 gcc/7.5.0 cudnn/7.6.5.32 nccl/2.5.6
conda create -n cupy-env python=3.7 numpy six fastrlock
source activate cupy-env
CFLAGS="-I$SCINET_CUDNN_ROOT/include -I$SCINET_NCCL_ROOT/include -I$SCINET_CUDA_ROOT/include" LDFLAGS="-L$SCINET_CUDNN_ROOT/lib64 -L$SCINET_NCCL_ROOT/lib" CUDA_PATH=$SCINET_CUDA_ROOT pip install cupy
#building/installing CuPy will take a few minutes
</pre>

== IBM Watson Machine Learning Community Edition (PowerAI) ==
[https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/ IBM Watson Machine Learning Community Edition (PowerAI)] contains many popular ML packages including TensorFlow, PyTorch, XGBoost and RAPIDS. It is distributed through IBM Conda channel. To install packages from PowerAI, user needs to specify IBM Conda channel when using Anaconda.
<pre>
module load anaconda3

conda create --name wmlce_env -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda <package_name> (e.g. powerai, tensorflow-gpu, keras, pytorch, powerai-rapids, py-xgboost-gpu, etc)

source activate wmlce_env
</pre>
*The WML CE Early Access Conda channel (https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/) makes new versions of frameworks available in advance of formal WML CE releases. Easy upgrade between packages in the main and Early Access channels is not guaranteed. Using a separate conda environment for Early Access packages is recommended.

== NAMD ==
[http://www.ks.uiuc.edu/Research/namd/ NAMD] is a parallel, object-oriented molecular dynamics code designed for high-performance simulation of large biomolecular systems.
=== v2.13 ===
<pre>
module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
</pre>
==== Running with one process per node====
An example of the job script (using 1 node, '''one process per node''', 32 CPU threads per process + 4 GPUs per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 1 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 32 +p $((32*SLURM_NTASKS)) stmv.namd
</pre>
==== Running with one process per GPU ====
NAMD may scale better if using '''one process per GPU'''. Please do your own benchmark.
An example of the job script (using 1 node, '''one process per GPU''', 8 CPU threads per process):
<pre>
#!/bin/bash
#SBATCH --time=20:00
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH -p compute_full_node

module load cuda/10.2.89 gcc/7.5.0 fftw/3.3.8 spectrum-mpi/10.3.1 namd/2.13
scontrol show hostnames > nodelist-$SLURM_JOB_ID

`which charmrun` -npernode 4 -hostfile nodelist-$SLURM_JOB_ID `which namd2` +setcpuaffinity +pemap 0-127:4 +idlepoll +ppn 8 +p $((8*SLURM_NTASKS)) stmv.namd
</pre>

== PyTorch ==
=== Installing from IBM Conda Channel ===
The easiest way to install PyTorch on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install PyTorch using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n pytorch_env python=3.7
source activate pytorch_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ pytorch
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== RAPIDS ==
The [https://rapids.ai RAPIDS] is a suite of open source software libraries that gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. The RAPIDS data science framework includes a collection of libraries: '''cuDF(GPU DataFrames)''', '''cuML(GPU Machine Learning Algorithms)''', '''cuStrings(GPU String Manipulation)''', etc.

=== Installing from IBM Conda Channel ===
The easiest way to install RAPIDS on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install powerai-rapids using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n rapids_env python=3.7
source activate rapids_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ powerai-rapids
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

== TensorFlow and Keras ==
=== Installing from IBM Conda Channel ===
The easiest way to install TensorFlow and Keras on Mist is using IBM's Conda channel. User needs to prepare a conda environment with Python 3.6 or 3.7 and install TensorFlow-gpu using IBM's Conda channel.
<pre>
module load anaconda3
conda create -n tf_env python=3.7
source activate tf_env
conda install -c https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/ tensorflow-gpu
</pre>
Once the installation finishes, please clean the cache:
<pre>
conda clean -y --all
rm -rf $HOME/.conda/pkgs/*
</pre>

= Testing and debugging =
You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.
* Small test jobs can be run on the login node. Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than one gpu and a few cores.

* Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
mist-login01:~$ debugjob --clean -g G
where G is the number of gpus, If G=1, this gives an interactive session for 2 hours, whereas G=4 gets you a single node with 4 gpus for 30 minutes, and with G=8 (the maximum) gets you 2 nodes each with 4 gpus for 30 minutes. The <tt>--clean</tt> argument is optional but recommended as it will start the session without any modules loaded, thus mimicking more closely what happens when you submit a job script.

= Submitting jobs =
Once you have compiled and tested your code or workflow on the Mist login nodes, and confirmed that it behaves correctly, you are ready to submit jobs to the cluster. Your jobs will run on some of Mist's 53 compute nodes. When and where your job runs is determined by the scheduler.

Mist uses SLURM as its job scheduler. It is configured to allow only '''Single-GPU jobs''' and '''Full-node jobs (4 GPUs per node)'''.

You submit jobs from a login node by passing a script to the sbatch command:

mist-login01:scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course. In most cases, you should not submit from your $HOME directory, but rather, from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

Example job scripts can be found below.
Keep in mind:
* Scheduling is by single gpu or by full node, so you ask only 1 gpu or 4 gpus per node.
* Your job's maximum walltime is 24 hours.
* Jobs must write their output to your scratch or project directory (home is read-only on compute nodes).
* Compute nodes have no internet access.
* Your job script will not remember the modules you have loaded, so it needs to contain "module load" commands of all the required modules (see examples below).
== SOSCIP Users ==
If you are working on a SOSCIP project, please contact soscip-support@scinet.utoronto.ca to have your user account added to SOSCIP project accounts. SOSCIP users need to submit jobs with additional SLURM flag:
<pre>
#SBATCH -A <SOSCIP_PROJECT_ID>
</pre>
== Single-GPU job script ==
For a single GPU job, each will have a quarter of the node which is 1 GPU + 8/32 CPU Cores/Threads + ~58GB CPU memory. '''Users should never ask CPU or Memory explicitly.''' If running MPI program, user can set --ntasks to be the number of MPI ranks. It is suggested to use NVIDIA Multi-Process Service (MPS) if running multiple MPI ranks on one GPU.
<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH -A <SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate conda_env
python code.py ...
</pre>

== Full-node job script ==
Multi-GPU job should ask for a minimum of one full node. User need to specify "compute_full_node" partition in order to get all resource on a node.
*An example for a 2-node, 8-rank OpenMPI job: (Each rank binds to 1 GPU and 8 physical CPU cores in this case)
<pre>
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --gpus-per-node=4
#SBATCH --ntasks=8
#SBATCH --time=1:00:00
#SBATCH -p compute_full_node
#SBATCH -A <SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load cuda/10.2.89 gcc/7.5.0 openmpi/4.0.2

mpirun -bind-to core -map-by slot:PE=8 -report-bindings ./program
</pre>

Main Page

2020-02-11T20:16:35Z

Nolta: /* System Status */

Globus

2019-11-21T20:00:31Z

Nolta:

Globus is a service for fast, reliable, secure data movement. Designed specifically for researchers, Globus has an easy-to-use interface with background monitoring features that automate the management of file transfers between any two resources, whether they are at Compute Canada, another supercomputing facility, a campus cluster, lab server, desktop or laptop.

__TOC__

== Copying files between Compute Canada sites using the Globus web interface ==

The procedure of using the [http://globus.computecanada.ca/ Globus web interface] for transferring data between different Compute Canada sites ("endpoints" in Globus), including those at SciNet, is well described in the [https://docs.computecanada.ca/wiki/Globus Compute Canada Globus documentation].

At SciNet, there are three endpoints:

=== <tt>computecandada#niagara</tt> ===

This endpoint gives access to the files on your Niagara $HOME. To get easy access to your files on your $SCRATCH in the Globus web interface, it can be helpful to add a so-called softlink in your home directory to the scratch directory, by issuing the following command on the command line on Niagara (once): <tt> $ ln -sn $SCRATCH $HOME/scratch</tt> A similar soft-link can be useful for your $PROJECT directory, if you have one on Niagara.

=== <tt>computecanada#hpss</tt> ===

This endpoint gives access to your [[HPSS]] space, if you have access to it. [[HPSS]] is a tape-backed hierarchical storage system that provides a significant portion of the allocated storage space at SciNet. The Globus endpoint is not the only way to interact with HPSS, and may not be appropriate method for your use case. Users should read the [[HPSS]] page before using this endpoint.

=== <tt>computecanada#bgq</tt> ===

This endpoint gives access to the files on your SciNet SOSCIP $HOME, if you have access to one of the SOSCIP systems hosted at SciNet, i.e., the [[BGQ]] or the [[SOSCIP GPU]] cluster. To get easy access in the Globus web interface to your SOSCIP $SCRATCH files, it can be helpful to add a so-called softlink in your home directory to your scratch directory, by issuing the following command on the command line on Niagara (once): <tt> $ ln -sn $SCRATCH $HOME/scratch</tt> 

== Copying files between Compute Canada sites and your personal computer using the Globus web interface ==

The procedure of using the [http://globus.computecanada.ca/ Globus web interface] for transferring data between Compute Canada sites (including those at SciNet) and your own personal computer, is well described in the [https://docs.computecanada.ca/wiki/Globus Compute Canada Globus documentation].

Essentially, you create an "endpoint for your personal computer", then you can use the web interface to transfer to one of the SciNet endpoints listed above.

== Copying Files to Niagara From the Linux Command-line ==

=== Step 1: Install globus CLI ===

Requires python 2.7. On the machine you're transferring from:

$ virtualenv venv-globus
$ source ./venv-globus/bin/activate
$ pip install globus-cli

=== Step 2: Login to globus ===

$ globus login

You should see:

Please log into Globus here:
---------------------------
https://auth.globus.org/v2/oauth2/...
---------------------------

Enter the resulting Authorization Code here:

Visit the URL in a web browser, choose "Compute Canada" as your organization, and enter your Niagara username/password.

=== Step 3: Create a personal endpoint ===

$ globus endpoint create --personal my-endpoint-name

Replace <code>my-endpoint-name</code> with a name of your choice.

You should see:

Message: Endpoint created successfully
Endpoint ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Setup Key: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy

Save this info as we'll need it later.

=== Step 4: Get Globus Connect Personal ===

$ wget https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz
$ tar -xzf globusconnectpersonal-latest.tgz
$ cd globusconnectpersonal-x.y.z

=== Step 5: Setup your endpoint ===

$ ./globusconnectpersonal -setup yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy

replacing "yyyy..." with the Setup Key from 'globus endpoint create'. You should see something like:

Configuration directory: /home/username/.globusonline/lta
Contacting relay.globusonline.org:2223
Done!

=== Step 6: Configure your endpoint (optional) ===

By default globus only allows transfers to/from your home directory.
Edit <code>~/.globusonline/lta/config-paths</code> and add a line for any other directories you need, e.g.:

/path/to/data/,0,1

=== Step 7: Start Globus Connect ===

$ ./globusconnectpersonal -start &

=== Step 8: Set some convenience variables ===

$ my_endpoint="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
$ niagara_endpoint="77506016-4a51-11e8-8f88-0a6d4e044368"

replacing "xxxx..." with the Endpoint ID from 'globus endpoint create'.

=== Step 9: Activate the Niagara endpoint ===

$ globus endpoint activate --myproxy --myproxy-lifetime=1000 $niagara_endpoint

=== Step 10: Start a transfer ===

$ globus transfer --recursive $my_endpoint:/path/to/data $niagara_endpoint:/scratch/g/group/username/data

Main Page

2019-11-19T17:40:27Z

Nolta: /* System Status */