SciNet Users Documentation - User contributions [en]

Main Page

2025-12-31T17:40:05Z

Dgruner: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up | OnDemand|Open_OnDemand_Quickstart}}
|{{Up | Globus |Globus}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Wed Dec 31, 2025, 12:40 pm:''' We believe the problem has now been resolved. Please let us know if you still experience login problems or aborted jobs.

'''Tue Dec 30, 2025, 2:10 pm:''' We are experiencing problems with authentication, resulting in failed logins, OOD errors, and aborted jobs (with "prolog error"). Please bear with us, as we are very short-staffed during the holiday break. We will post updates here.

'''Tue Dec 3, 2025, 11:30 am:''' Open OnDemand is fully operational again.

'''Sat Nov 29, 2025, 00:40 am:''' There has been a problem with the water chiller. Some systems are offline.

'''Wed Nov 5, 2025, 12:55 pm:''' Balam is back online.

'''Wed Nov 5, 2025, 10:00 am:''' Open OnDemand is back online.

'''Tue Nov 4, 2025, 11:00 pm:''' Most of the work is done, data movers, Globus, and HPSS are back online. Remaining services will be worked on tomorrow.

'''Tue Nov 4, 2025, 8:30 am:''' Scheduled network maintenance. Trillium cluster is *not* affected.

'''Tue Oct 21, 2025, 17:30 am:''' Balam maintenance finished.

'''Tue Oct 21, 2025, 7:00 am:''' Balam maintenance day.

'''Thu Oct 15, 2025, 3:55 pm:''' Trillium inbound connections through trillium.alliancecan.ca or trillium.scinet.utoronto.ca are working again.

'''Thu Oct 15, 2025, 3:05 pm:''' Trillium is experiencing external network issues for both incoming traffic. Please try: ssh USERNAME@tri-login01.scinet.utoronto.ca in the meantime.

'''Thu Oct 06, 2025, 8:00 pm:''' HPSS is fully functional. You may submit archive jobs from trillium login nodes, datamovers and robots.

'''Thu Oct 03, 2025, 6:30 pm:''' HPSS is back online, and already accessible via alliancecan#hpss Globus endpoint. Directory tree now follows the other Alliance clusters. We're still working on job submission via Slurm

'''Thu Oct 01, 2025, 0:00 am:''' Niagara compute nodes are now unavailable for regular users. The login nodes will remain available for a while to allow a few last data transfers, although transfers from the Niagara file systems to Trillium are best done on nia-dm1.scinet.utoronto.ca.

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2025-12-30T19:11:36Z

Dgruner: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up | OnDemand|Open_OnDemand_Quickstart}}
|{{Up | Globus |Globus}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Tue Dec 30, 2025, 2:10 pm:''' We are experiencing problems with authentication, resulting in failed logins, OOD errors, and aborted jobs (with "prolog error"). Please bear with us, as we are very short-staffed during the holiday break. We will post updates here.

'''Tue Dec 3, 2025, 11:30 am:''' Open OnDemand is fully operational again.

'''Sat Nov 29, 2025, 00:40 am:''' There has been a problem with the water chiller. Some systems are offline.

'''Wed Nov 5, 2025, 12:55 pm:''' Balam is back online.

'''Wed Nov 5, 2025, 10:00 am:''' Open OnDemand is back online.

'''Tue Nov 4, 2025, 11:00 pm:''' Most of the work is done, data movers, Globus, and HPSS are back online. Remaining services will be worked on tomorrow.

'''Tue Nov 4, 2025, 8:30 am:''' Scheduled network maintenance. Trillium cluster is *not* affected.

'''Tue Oct 21, 2025, 17:30 am:''' Balam maintenance finished.

'''Tue Oct 21, 2025, 7:00 am:''' Balam maintenance day.

'''Thu Oct 15, 2025, 3:55 pm:''' Trillium inbound connections through trillium.alliancecan.ca or trillium.scinet.utoronto.ca are working again.

'''Thu Oct 15, 2025, 3:05 pm:''' Trillium is experiencing external network issues for both incoming traffic. Please try: ssh USERNAME@tri-login01.scinet.utoronto.ca in the meantime.

'''Thu Oct 06, 2025, 8:00 pm:''' HPSS is fully functional. You may submit archive jobs from trillium login nodes, datamovers and robots.

'''Thu Oct 03, 2025, 6:30 pm:''' HPSS is back online, and already accessible via alliancecan#hpss Globus endpoint. Directory tree now follows the other Alliance clusters. We're still working on job submission via Slurm

'''Thu Oct 01, 2025, 0:00 am:''' Niagara compute nodes are now unavailable for regular users. The login nodes will remain available for a while to allow a few last data transfers, although transfers from the Niagara file systems to Trillium are best done on nia-dm1.scinet.utoronto.ca.

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

S4H

2025-12-02T21:43:03Z

Dgruner:

= Introduction =
S4H (formerly SciNet4Health) is our secure computing environment pilot, providing users with the ability to run [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium] jobs on confidential data. This subsystem is comprised of a dedicated login node and a storage appliance, but it is highly integrated with Trillium. Security concerns are addressed by

* Hardened access
* Encryption at rest
* Group isolation
* Data egress control (optional)

Usage of S4H is by request only. Access must be requested by a principal investigator (PI) on behalf of their group members (i.e. sponsored users on CCDB).

= Policies =
Each user is assigned one of three policies:

* '''Permissive:''' the user may connect to the login node using SSH from pre-approved source IP addresses, and has unrestricted internet access from the login node
* '''Restrictive:''' the user may connect to the login node using SSH from pre-approved source IP addresses, but internet access from the login node is restricted
* '''Prohibitive:''' the user may only connect to the login node using a remote desktop client program from pre-approved source IP addresses, and internet access from the login node is restricted

If you don't know what policy you belong to, you should ask your PI.

= Login =
== Direct ==
Users with permission to connect directly to the login node (permissive and restrictive policies) should first make sure that they are able to login to Trillium (i.e. they have uploaded an SSH public key to, and set up second factor authentication on CCDB). If access to Trillium is successful, use the same username and SSH key to login to the following address:

<pre>
s4h.scinet.utoronto.ca
</pre>

You should be prompted for the second factor, like in Trillium.

The connection must be made from one of the '''IP addresses pre-approved by the PI''' for that user (e.g. a workstation or a jump host in your lab).

== Through the graphical gateway ==
Users with permission to connect through the graphical gateway should use an RDP-enabled remote desktop client and login to the following address using their CCDB username and password:

<pre>
s4h-ggw.scinet.utoronto.ca
</pre>

It is recommended to set the display resolution to 1600×900 if the resolution is not picked up automatically by the program.

Additionally, a "pre-login" step has to be performed. In this step, an SSH ''agent'' must be forwarded to the above address. The graphical gateway asks the user's SSH agent program to perform the authentication (i.e. the workstation where the user is connecting from has the private key and it has been added to the agent). In a sense this is a 3-factor authentication: one needs the password, the SSH private key, and have either a YubiKey or the Duo mobile app registered with CCDB. Here is an example of this process:

<pre>
eval $(ssh-agent)
ssh-add /home/alice/.ssh/ccdb_ed25519
ssh -T -A alice@s4h-ggw.scinet.utoronto.ca
</pre>

Note that no shell access is expected after the <code>ssh</code> command; but the window on the remote desktop program should now prompt for the YubiKey passcode or Duo mobile app push. Once that is done as well, the SSH client will print _Login successful_ and quit.

The connection must be made from one of the '''IP addresses pre-approved by the PI''' for that user (e.g. a workstation or a jump host in your lab).

= Storage =
== Directories ==
Trillium file systems are accessible via their usual paths but are read-only on S4H (to prevent accidentally saving sensitive data there). Instead, home, scratch, and project spaces are provided on alternative paths under <code>/s4h</code> (indicating the encrypted storage appliance). If the user "alice" belongs to the group "def-bob" on S4H, their home directory (which can be expanded from <code>~</code> or <code>$HOME</code>) will be located in <code>/s4h/def-bob/home/alice</code> and similarly their scratch directory (can be expanded from <code>$SCRATCH</code>) will be in <code>/s4h/def-bob/scratch/alice</code>. The project directory is <code>/s4h/def-bob/project</code>, users may create their own directories there as needed.

The environment variables <code>$TRIHOME</code>, <code>$TRISCRATCH</code>, and <code>$TRIPROJECT</code> expand to the corresponding file system paths for Trillium (as noted above, they are read-only on S4H)

== Data transfer ==
For users in the permissive and restrictive policies, please use an SSH-based program (such as <code>scp</code> or <code>rsync</code>) to transfer data directly in and out of the S4H login node. There is no dedicated datamover for S4H.

Under the prohibitive policy, users may not transfer sensitive data in and out of S4H. They may only upload non-sensitive data to Trillium (storage not encrypted at rest) where they can be accessed from S4H. For egress purposes, the PI should designate at least one user (could be themselves) that is not under the prohibitive policy. Other users in the group could share files for egress with the designated user (e.g. by putting them in the group's project directory).

== Data policies ==
It is important to understand that:
* Within a group, file access is managed by traditional POSIX permissions and access-control lists, like on Trillium. In case users in a group are working on separate sub-projects where there should not be mutual access, it is their responsibility to make sure that permissions are set up correctly.
* There is no way to facilitate cross-group file sharing of sensitive files on S4H; each group has a different encryption key and the system is set up so that a compute node can only use one key at a time.
* No backup is provided for encrypted storage; deletion is irreversible. This ensure that data are securely disposed of in compliance with a provision found in many data sharing agreements.

= Software =
Same as Trillium.

Note that you may use software (including, for example, Python virtual environments) that you installed in your Trillium file systems on S4H (but not vice versa). This could be useful for users under the restrictive or prohibitive policy, that may otherwise have difficulty installing software in their encrypted storage spaces.

= Submitting jobs =
This is largely the same as Trillium. Note however that job metadata are not kept confidential! In particular, the submitting user, work directory, job name, comment, and command should be ''considered public information'' and the users must not include any sensitive information in these.

Main Page

2025-07-05T02:06:46Z

Dgruner: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Partial | Niagara|Niagara_Quickstart}}
|{{Partial | Mist|Mist}}
|{{Up | Teach|Teach}}
|{{Up | Rouge|Rouge}}
|-
|{{Up | OnDemand|Open_OnDemand_Quickstart}}
|{{Up | Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up | File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up | External Network|Niagara_Quickstart#Logging_in}}
|{{Up | Globus |Globus}}
|-
|{{Up | Balam|Balam}}
|{{Up | Cvmfs|Using_modules}}
|}

'''July 9, 2025:''' The [[Teach]] cluster will be unavailable for the day for network maintenance.

'''July 4, 2025:''' Open OnDemand is back up.

'''July 4, 2025:''' Open OnDemand is down. We are investigating.

'''June 25, 2025, 7:15 PM EDT:''' The [[Teach]] cluster's scheduler is up again.

'''June 25, 2025, 4:30 PM EDT:''' The [[Teach]] cluster's scheduler is down. We are investigating.

'''April 30, 2025, 9:30 AM EDT:''' The [[Teach]] cluster is available again.

'''April 30, 2025:''' The [[Teach]] cluster will be unavailable from 8:00 am to about 12:00 noon for file system maintenance.

'''April 1, 2025:''' The Jupyter Hub has been replaced by SciNet's [[Open OnDemand Quickstart|Open OnDemand service]].

'''March 1, 2025:''' As of March 1st scratch purging is suspended until after Trillium comes online.

'''January 6, 2025:''' As part of the installation of the new computing cluster Trillium, there is now a permanent reduction in computing capacity of Niagara to 50% and of Mist to 35%.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2025-07-05T02:05:58Z

Dgruner: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Partial | Niagara|Niagara_Quickstart}}
|{{Partial | Mist|Mist}}
|{{Up | Teach|Teach}}
|{{Up | Rouge|Rouge}}
|-
|{{Up | OnDemand|Open_OnDemand_Quickstart}}
|{{Up | Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up | File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up | External Network|Niagara_Quickstart#Logging_in}}
|{{Up | Globus |Globus}}
|-
|{{Up | Balam|Balam}}
|{{Up | Cvmfs|Using_modules}}
|}

'''July 9, 2025:''' The [[Teach]] cluster will be unavailable for the day for network maintenance.

'''July 4, 2025:''' Open OnDemand is down. We are investigating.

'''June 25, 2025, 7:15 PM EDT:''' The [[Teach]] cluster's scheduler is up again.

'''June 25, 2025, 4:30 PM EDT:''' The [[Teach]] cluster's scheduler is down. We are investigating.

'''April 30, 2025, 9:30 AM EDT:''' The [[Teach]] cluster is available again.

'''April 30, 2025:''' The [[Teach]] cluster will be unavailable from 8:00 am to about 12:00 noon for file system maintenance.

'''April 1, 2025:''' The Jupyter Hub has been replaced by SciNet's [[Open OnDemand Quickstart|Open OnDemand service]].

'''March 1, 2025:''' As of March 1st scratch purging is suspended until after Trillium comes online.

'''January 6, 2025:''' As part of the installation of the new computing cluster Trillium, there is now a permanent reduction in computing capacity of Niagara to 50% and of Mist to 35%.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2024-10-21T21:11:45Z

Dgruner: /* System Status */

Main Page

2024-08-02T02:01:20Z

Dgruner: /* System Status */

Main Page

2024-08-02T01:42:17Z

Dgruner: /* System Status */

Gurobi

2024-06-14T19:58:02Z

Dgruner: /* Using Gurobi */

The [http://www.gurobi.com/ Gurobi] linear programming solver is installed on the Niagara software stack.

"Given a set of linear inequality/equality constraints, Ax>=b, where A is a matrix and x & b are vectors, what is the set of variables x (within a given range) that maximizes/minimizes a target objective function f(x)?"

Such a model is very common in scientific computation, engineering, and business. If the variables x are (partially) limited to integers, this becomes Mixed Integer Programming (MIP) which is a much more sophisticated problem. Gurobi, along with other solvers (such as "linprog" and "intlinprog" in [[MATLAB]]) can solve such LP/MIP problems efficiently. And for Gurobi, for instance, efficient multi-threading (and even distributed computation, depending on the license) capabilities are also implemented for parallelism and easy scaling for large models.

=Getting a license=
The University of Toronto has a free academic license to use Gurobi. Access to the license is granted by loading the Gurobi module.

=Running using the Niagara installation=

==Gurobi 11.0.1==
To access commercial modules on Niagara one must invoke the 'module use' command.

module load NiaEnv/2022a
module use /scinet/niagara/software/commercial/modules
module load gurobi/11.0.1

==Using Gurobi==

To use Gurobi, one only needs to include "gurobi_c++.h" in the source file, and use the compilation/linking flags:

CXXLIB=-L ${SCINET_GUROBI_LIB} -lgurobi_g++8.5 -lgurobi110 -fopenmp
CXXINC=-I ${SCINET_GUROBI_INC} -fopenmp

The actual documentation for using Gurobi's API can be found [http://www.gurobi.com/documentation/11.0 here].

==Running Gurobi==
Example submission script for a job running on 1 node, with max walltime of 11 hours:

<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=40
#SBATCH --time=11:00:00
#SBATCH --job-name test

module load NiaEnv/2022a
module use /scinet/niagara/software/commercial/modules
module load gurobi/11.0.1

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# If you are using OpenMP
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

./mycode
</source>

Gurobi

2024-06-14T19:55:41Z

Dgruner: /* Running Gurobi */

The [http://www.gurobi.com/ Gurobi] linear programming solver is installed on the Niagara software stack.

"Given a set of linear inequality/equality constraints, Ax>=b, where A is a matrix and x & b are vectors, what is the set of variables x (within a given range) that maximizes/minimizes a target objective function f(x)?"

Such a model is very common in scientific computation, engineering, and business. If the variables x are (partially) limited to integers, this becomes Mixed Integer Programming (MIP) which is a much more sophisticated problem. Gurobi, along with other solvers (such as "linprog" and "intlinprog" in [[MATLAB]]) can solve such LP/MIP problems efficiently. And for Gurobi, for instance, efficient multi-threading (and even distributed computation, depending on the license) capabilities are also implemented for parallelism and easy scaling for large models.

=Getting a license=
The University of Toronto has a free academic license to use Gurobi. Access to the license is granted by loading the Gurobi module.

=Running using the Niagara installation=

==Gurobi 11.0.1==
To access commercial modules on Niagara one must invoke the 'module use' command.

module load NiaEnv/2022a
module use /scinet/niagara/software/commercial/modules
module load gurobi/11.0.1

==Using Gurobi==

To use Gurobi, one only needs to include "gurobi_c++.h" in the source file, and use the compilation/linking flags:

CXXLIB=-L ${SCINET_GUROBI_LIB} -lgurobi_g++5.2 -lgurobi75 -fopenmp
CXXINC=-I ${SCINET_GUROBI_INC} -fopenmp

The actual documentation for using Gurobi's API can be found [http://www.gurobi.com/documentation/7.5 here].

==Running Gurobi==
Example submission script for a job running on 1 node, with max walltime of 11 hours:

<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=40
#SBATCH --time=11:00:00
#SBATCH --job-name test

module load NiaEnv/2022a
module use /scinet/niagara/software/commercial/modules
module load gurobi/11.0.1

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# If you are using OpenMP
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

./mycode
</source>

Gurobi

2024-06-14T19:54:43Z

Dgruner: /* Gurobi 9.5.1 */

The [http://www.gurobi.com/ Gurobi] linear programming solver is installed on the Niagara software stack.

"Given a set of linear inequality/equality constraints, Ax>=b, where A is a matrix and x & b are vectors, what is the set of variables x (within a given range) that maximizes/minimizes a target objective function f(x)?"

Such a model is very common in scientific computation, engineering, and business. If the variables x are (partially) limited to integers, this becomes Mixed Integer Programming (MIP) which is a much more sophisticated problem. Gurobi, along with other solvers (such as "linprog" and "intlinprog" in [[MATLAB]]) can solve such LP/MIP problems efficiently. And for Gurobi, for instance, efficient multi-threading (and even distributed computation, depending on the license) capabilities are also implemented for parallelism and easy scaling for large models.

=Getting a license=
The University of Toronto has a free academic license to use Gurobi. Access to the license is granted by loading the Gurobi module.

=Running using the Niagara installation=

==Gurobi 11.0.1==
To access commercial modules on Niagara one must invoke the 'module use' command.

module load NiaEnv/2022a
module use /scinet/niagara/software/commercial/modules
module load gurobi/11.0.1

==Using Gurobi==

To use Gurobi, one only needs to include "gurobi_c++.h" in the source file, and use the compilation/linking flags:

CXXLIB=-L ${SCINET_GUROBI_LIB} -lgurobi_g++5.2 -lgurobi75 -fopenmp
CXXINC=-I ${SCINET_GUROBI_INC} -fopenmp

The actual documentation for using Gurobi's API can be found [http://www.gurobi.com/documentation/7.5 here].

==Running Gurobi==
Example submission script for a job running on 1 node, with max walltime of 11 hours:

<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=40
#SBATCH --time=11:00:00
#SBATCH --job-name test

module use /scinet/niagara/software/commercial/modules
module load gurobi/7.5.2

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# If you are using OpenMP
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

./mycode
</source>

Main Page

2024-05-29T17:55:35Z

Dgruner: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down |Niagara|Niagara_Quickstart}}
|{{Down |Mist|Mist}}
|{{Down |Teach|Teach}}
|{{Down |Rouge|Rouge}}
|-
|{{Down |Jupyter Hub|Jupyter_Hub}}
|{{Down |Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Down |File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Down |Burst Buffer|Burst_Buffer}}
|-
|{{Down |HPSS|HPSS}}
|{{Down |Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Down |External Network|Niagara_Quickstart#Logging_in}}
|{{Down |Globus |Globus}}
|-
|{{Down |Balam|Balam}}
|{{Down |CCEnv|Using_modules}}
|}

'''Wednesday May 29, 2 PM EDT''' Electricians are checking and testing all junction boxes and connectors under the raised floor for safety. Some systems are expected to be back up later today (storage, login nodes), and compute systems will be powered up as soon as it is deemed safe.

'''Tuesday May 28, 3 PM EDT''' Cleaning crews are at the datacentre, to pump the water and install dryers. Once the floors are dry, we need to inspect all electrical boxes to ensure safety. We do not expect to have a fully functional datacentre before Thursday, although we hope to be able to turn on the storage and login nodes sometime tomorrow, if circumstances permit. Apologies, and thank you for your patience.

'''Tuesday May 28, 7 AM EDT''' A water mains break outside our datacentre has caused extensive flooding, and all systems have been shut down preventatively.

'''Friday May 17, 10 PM EDT - Saturday May 18, 2 AM EDT:''' The external network will be unavailable for maintenance. Running and queued jobs on the systems will not be affected.

'''Tuesday May 14, 6:45 PM EDT:''' All systems are recovered now.

'''Tuesday May 14, 5 PM EDT:''' Power loss at the datacentre resulted in loss of cooling. Systems are being restored.

'''Friday May 3, 10 PM EDT - Saturday May 4, 2 AM EDT:''' The external network will be unavailable for maintenance. Running and queued jobs on the systems will not be affected.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2024-05-28T19:17:20Z

Dgruner: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down |Niagara|Niagara_Quickstart}}
|{{Down |Mist|Mist}}
|{{Down |Teach|Teach}}
|{{Down |Rouge|Rouge}}
|-
|{{Down |Jupyter Hub|Jupyter_Hub}}
|{{Down |Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Down |File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Down |Burst Buffer|Burst_Buffer}}
|-
|{{Down |HPSS|HPSS}}
|{{Down |Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Down |External Network|Niagara_Quickstart#Logging_in}}
|{{Down |Globus |Globus}}
|-
|{{Down |Balam|Balam}}
|{{Down |CCEnv|Using_modules}}
|}

'''Tuesday May 28, 3 PM EDT''' Cleaning crews are at the datacentre, to pump the water and install dryers. Once the floors are dry, we need to inspect all electrical boxes to ensure safety. We do not expect to have a fully functional datacentre before Thursday, although we hope to be able to turn on the storage and login nodes sometime tomorrow, if circumstances permit. Apologies, and thank you for your patience.

'''Tuesday May 28, 7 AM EDT''' A water mains break outside our datacentre has caused extensive flooding, and all systems have been shut down preventatively.

'''Friday May 17, 10 PM EDT - Saturday May 18, 2 AM EDT:''' The external network will be unavailable for maintenance. Running and queued jobs on the systems will not be affected.

'''Tuesday May 14, 6:45 PM EDT:''' All systems are recovered now.

'''Tuesday May 14, 5 PM EDT:''' Power loss at the datacentre resulted in loss of cooling. Systems are being restored.

'''Friday May 3, 10 PM EDT - Saturday May 4, 2 AM EDT:''' The external network will be unavailable for maintenance. Running and queued jobs on the systems will not be affected.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2024-05-28T10:58:14Z

Dgruner: /* System Status */

Main Page

2024-05-28T10:56:11Z

Dgruner: /* System Status */

Main Page

2024-05-14T21:30:54Z

Dgruner: /* System Status */

Balam

2023-12-19T22:24:49Z

Dgruner: /* Specifications */

{{Infobox Computer
|image=[[File:Balam.jpeg|center|300px|thumb]]
|name=Balam
|installed=October 2023
|operatingsystem= Linux (Rocky 9.2)
|loginnode= balam-login01
|nnodes=10
|gpuspernode=4 A100-40GB
|rampernode=1 TB
|corespernode=64
|interconnect=Infiniband
|vendorcompilers=cuda/intel/gcc
|queuetype=slurm
}}

= Specifications=

The Balam cluster is owned by the [https://acceleration.utoronto.ca Acceleration Consortium at the University of Toronto], and hosted at SciNet The cluster consists 10 x86_64 nodes each with two Intel Xeon(R) Platinum 8358 32-core CPUs running at 2.6GHz with 1 TB of RAM and four NVIDIA A100 GPUs per node.

The nodes are interconnected with Infiniband for internode communications and disk I/O to the SciNet Niagara file systems. In total this cluster contains 640 CPU cores and 40 GPUs.

Access is available only to those affiliated with the Acceleration Consortium. Support requests should be sent to '''balam-support@scinet.utoronto.ca'''.

= Getting started on Balam =

Balam can be accessed directly.
<pre>
ssh -Y MYCCUSERNAME@balam.scinet.utoronto.ca
</pre>

Or, the Balam login node '''balam-login01''' can be accessed via the Niagara cluster.
<pre>
ssh -Y MYCCUSERNAME@niagara.scinet.utoronto.ca
ssh -Y balam-login01
</pre>

== Storage ==

The filesystem for Balam is currently shared with Niagara cluster. See [https://docs.scinet.utoronto.ca/index.php/Niagara_Quickstart#Your_various_directories Niagara Storage] for more details.

= Loading software modules =

You have two options for running code on Balam: use existing software, or compile your own. This section focuses on the former.

Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]].

Common module subcommands are:

* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list loadable software packages.
* <code>module list</code>: list loaded modules.

Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a MODULE_MODULENAME_PREFIX environment variable, which can be used to access commonly needed software directories, such as /include and /lib.

There are handy abbreviations for the module commands. <code>ml</code> is the same as <code>module list</code>, and <code>ml <module-name></code> is the same as <code>module load <module-name></code>.

== Software stacks: BalamEnv and CCEnv ==

On Balam, there are two available software stacks:

=== BalamEnv ===

A software stack with [[Modules specific to Balam]] tuned and compiled for this machine. This stack is available by default, but if not, can be reloaded with
<pre>module load BalamEnv</pre>
This loads the default (set of modules), which is currently the 2023a epoch.

No modules are loaded by default on Balam except BalamEnv.

=== CCEnv ===

The same [https://docs.alliancecan.ca/wiki/Modules software stack available on {{Alliance}}'s General Purpose clusters] too, with:
<pre>module load CCEnv</pre>
Or, if you want the same default modules loaded as on Béluga and Narval, then do
<pre>module load CCEnv StdEnv</pre>
or, if you want the same default modules loaded as on Cedar and Graham, do
<pre>module load CCEnv arch/avx2 StdEnv/2020</pre>

= Available compilers and interpreters =

* In the BalamEnv, the <tt>cuda</tt> module has to be loaded first for GPU software.
* To compile mpi code, you must additionally load an <tt>openmpi</tt> module.

=== CUDA ===

The current installed CUDA versions are 11.8.0 and 12.3.1
<pre>
module load cuda/<version>
</pre>

The current NVIDIA driver version is 535.104.12. Use '''nvidia-smi -a''' for full details.

===Other Compilers and Tools ===

Other available compiler modules are:

<code>gcc/12.3.0</code> GNU Compiler Collection, compatible with CUDA/12.3

<code>gcc/13.2.0</code> GNU Compiler Collection, incompatible with CUDA/12.3

<code>intel/2023u1</code> Intel compiler suite

=== OpenMPI ===

<tt>openmpi/5.0.0</tt> module is available once <tt>gcc/13.2.0</tt> is loaded.

= Testing and debugging =



= Submitting jobs =

Main Page

2023-03-17T13:54:47Z

Dgruner: /* System Status */

Main Page

2023-02-10T00:25:42Z

Dgruner: /* System Status */

Main Page

2022-12-21T17:06:20Z

Dgruner: /* System Status */

Gurobi

2022-10-19T15:06:25Z

Dgruner:

The [http://www.gurobi.com/ Gurobi] linear programming solver is installed on the Niagara software stack.

"Given a set of linear inequality/equality constraints, Ax>=b, where A is a matrix and x & b are vectors, what is the set of variables x (within a given range) that maximizes/minimizes a target objective function f(x)?"

Such a model is very common in scientific computation, engineering, and business. If the variables x are (partially) limited to integers, this becomes Mixed Integer Programming (MIP) which is a much more sophisticated problem. Gurobi, along with other solvers (such as "linprog" and "intlinprog" in [[MATLAB]]) can solve such LP/MIP problems efficiently. And for Gurobi, for instance, efficient multi-threading (and even distributed computation, depending on the license) capabilities are also implemented for parallelism and easy scaling for large models.

=Getting a license=
The University of Toronto has a free academic license to use Gurobi. Access to the license is granted by loading the Gurobi module.

=Running using the Niagara installation=

==Gurobi 9.5.1==
To access commercial modules on Niagara one must invoke the 'module use' command.

module load NiaEnv/2019b
module use /scinet/niagara/software/commercial/modules
module load gurobi/9.5.1

==Using Gurobi==

To use Gurobi, one only needs to include "gurobi_c++.h" in the source file, and use the compilation/linking flags:

CXXLIB=-L ${SCINET_GUROBI_LIB} -lgurobi_g++5.2 -lgurobi75 -fopenmp
CXXINC=-I ${SCINET_GUROBI_INC} -fopenmp

The actual documentation for using Gurobi's API can be found [http://www.gurobi.com/documentation/7.5 here].

==Running Gurobi==
Example submission script for a job running on 1 node, with max walltime of 11 hours:

<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=40
#SBATCH --time=11:00:00
#SBATCH --job-name test

module use /scinet/niagara/software/commercial/modules
module load gurobi/7.5.2

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# If you are using OpenMP
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

./mycode
</source>

Singularity

2022-06-23T18:32:50Z

Dgruner:

Please see our Docker page.

Main Page

2022-05-23T20:46:46Z

Dgruner: /* System Status */

Main Page

2022-05-23T12:15:46Z

Dgruner: /* System Status */

Main Page

2022-05-09T11:27:49Z

Dgruner: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up |Niagara|Niagara_Quickstart}}
|{{Up |Mist|Mist}}
|{{Up |Teach|Teach}}
|{{Up |Rouge|Rouge}}
|-
|{{Up |Jupyter Hub|Jupyter_Hub}}
|{{Up |Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up |File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up |Burst Buffer|Burst_Buffer}}
|-
|{{Up |HPSS|HPSS}}
|{{Up |Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up |External Network|Niagara_Quickstart#Logging_in}}
|{{Up |Globus |Globus}}
|}

'''Fri May 6th, 2022, 11:35 am:''' HPSS scheduler upgrade also finished.

'''Thu May 5th, 2022, 7:45 pm:''' Upgrade of the scheduler has finished, with the exception of HPSS.

'''Thu May 5th, 2022, 7:00 am - 3:00 pm EDT (approx):''' Starting from 7:00 am EDT, an upgrade of the scheduler of the Niagara, Mist, and Rouge clusters will be applied. This requires the scheduler to be down for about 5-6 hours, and all compute and login nodes to be rebooted.
Jobs cannot be submitted during this maintenance, but jobs submitted beforehand will remain in the queue. For most of the time, the login nodes of the clusters will be available so that users may access their files on the home, scratch, and project file systems.

'''Monday May 2nd, 2022, 9:30 - 11:00 am EDT:''' the Niagara login nodes, the jupyter hub, and nia-datamover2 will get rebooted for updates. In the process, any login sessions will get disconnected, and servers on the jupyterhub will stop. Jobs in the Niagara queue will not be affected.

'''Tue Apr 26, 11:20 AM EDT:''' A Rolling update of the Mist cluster is taking a bit longer than expected, affecting logins to Mist.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[SSH#Two-Factor_authentication|Two-Factor Authentication]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2022-01-29T16:23:34Z

Dgruner: /* System Status */

Main Page

2022-01-28T12:38:28Z

Dgruner: /* System Status */

Main Page

2022-01-08T16:44:41Z

Dgruner: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up |Niagara|Niagara_Quickstart}}
|{{Up |Mist|Mist}}
|{{Up |Teach|Teach}}
|{{Up |Rouge|Rouge}}
|-
|{{Down |Jupyter Hub|Jupyter_Hub}}
|{{Up |Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up |File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up |Burst Buffer|Burst_Buffer}}
|-
|{{Up|HPSS|HPSS}}
|{{Up |Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up |External Network|Niagara_Quickstart#Logging_in}}
|{{Up |Globus |Globus}}
|}



Sat Jan 8 11:42 EST AM 2022 The emergency maintenance is complete. Systems are up and available.

Fri Jan 7 14:34 EST PM 2022 The SciNet shutdown is in progress. Systems are expected back on Saturday, Jan 8.

Emergency shutdown Friday January 7, 2022: An emergency shutdown of all SciNet to replace a crucial file system component is planned to take place on Friday January 7, 2022, starting at 8am EST, and will require at least 12 hours of downtime. Updates will be posted during the day.

Thu Jan 6 08:20 EST AM 2022 The SciNet filesystem is having issues. We are investigating.

Fri Dec 24 13:31 EST PM 2021 Please note the following scheduled network maintenance, which will result in loss of connectivity to the SciNet datacentre: Start time
Dec 29, 00:30 EST Estimated duration 4 hours and 30 minutes.

Mon Dec 20 4:29 EST PM 2021 Filesystem is back to normal.

Mon Dec 20 2:53 EST PM 2021 Filesystem problem - We are investigating.


{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH keys]]
* [[SSH Tunneling]]
* [[SSH#Two-Factor_authentication|Two-Factor Authentication]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2022-01-07T19:36:48Z

Dgruner: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down |Niagara|Niagara_Quickstart}}
|{{Down |Mist|Mist}}
|{{Down |Teach|Teach}}
|{{Down |Rouge|Rouge}}
|-
|{{Down |Jupyter Hub|Jupyter_Hub}}
|{{Down |Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Down |File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Down |Burst Buffer|Burst_Buffer}}
|-
|{{Down|HPSS|HPSS}}
|{{Down |Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Down |External Network|Niagara_Quickstart#Logging_in}}
|{{Down |Globus |Globus}}
|}



Fri Jan 7 14:34 EST PM 2022 The SciNet shutdown is in progress. Systems are expected back on Saturday, Jan 8.

Emergency shutdown Friday January 7, 2022: An emergency shutdown of all SciNet to replace a crucial file system component is planned to take place on Friday January 7, 2022, starting at 8am EST, and will require at least 12 hours of downtime. Updates will be posted during the day.

Thu Jan 6 08:20 EST AM 2022 The SciNet filesystem is having issues. We are investigating.

Fri Dec 24 13:31 EST PM 2021 Please note the following scheduled network maintenance, which will result in loss of connectivity to the SciNet datacentre: Start time
Dec 29, 00:30 EST Estimated duration 4 hours and 30 minutes.

Mon Dec 20 4:29 EST PM 2021 Filesystem is back to normal.

Mon Dec 20 2:53 EST PM 2021 Filesystem problem - We are investigating.


{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH keys]]
* [[SSH Tunneling]]
* [[SSH#Two-Factor_authentication|Two-Factor Authentication]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2022-01-07T13:09:44Z

Dgruner: /* System Status */

Main Page

2022-01-06T19:17:47Z

Dgruner: /* System Status */

Main Page

2022-01-06T18:47:47Z

Dgruner: /* System Status */

Main Page

2021-12-24T18:34:26Z

Dgruner: /* System Status */

Main Page

2021-11-06T00:22:08Z

Dgruner: /* System Status */

NAMD

2020-09-14T16:31:25Z

Dgruner: /* NAMD v2.14 */

====NAMD v2.14====

This is the NAMD version 2.14 Scalable Molecular Dynamics package.
NAMD_2.14_Linux-x86_64-ibverbs-smp.tar.gz binary tarball from TCBG.

It was built directly on top of ibverbs, plus smp, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

module load namd/2.14

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=2
P=$(($SLURM_NTASKS * 2))

charmrun ++verbose +p $P ++ppn $PPN ++nodelist $NODELIST $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

====NAMD v2.12====

This is the NAMD version 2.12 Scalable Molecular Dynamics package.

It was built directly on top of ibverbs, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

# Note that the module will likely be taken out of experimental mode at some point.
module load namd/.experimental-2.12-ibverbs-smp

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=4
P=$(($SLURM_NTASKS * 2))

charmrun ++verbose +p $P ++ppn $PPN ++nodelist $NODELIST $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

Main Page

2020-08-18T14:09:05Z

Dgruner: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down|Niagara|Niagara_Quickstart}}
|{{Down|HPSS|HPSS}}
|{{Down|Mist|Mist}}
|{{Down|Teach|Teach}}
|-
|{{Down|Jupyter Hub|Jupyter_Hub}}
|{{Down|Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Down|File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Down|Burst Buffer|Burst_Buffer}}
|-
|{{Down|Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Down|External Network|Niagara_Quickstart#Logging_in}}
|{{Down|Globus|Globus}}
|}


August 17, 2020, 4:00 PM EST: Unfortunately after taking the pump apart it was determined there was a more serious failure of the main drive shaft, not just the seal. As a new one will need to be sourced or fabricated we're estimating that it will take at least a few more days to get the part and repairs done to restore cooling. Sorry for the inconvenience.

August 15, 2020, 1:00 PM EST: Due to parts availablity to repair the failed pump and cooling system it is unlikely that systems will be able to be restored until Monday afternoon at the earliest.

August 15, 2020, 00:04 AM EST: A primary pump seal in the cooling infrastructure has blown and parts availability will not be able be determined until tomorrow. All systems are shut down as there is no cooling. If parts are available, systems may be back at the earliest late tomorrow. Check here for updates.

August 14, 2020, 21:04 AM EST: Tomorrow's /scratch purge has been postponed.

August 14, 2020, 21:00 AM EST: Staff at the datacenter. Looks like one of the pumps has a seal that is leaking badly.

August 14, 2020, 20:37 AM EST: We seem to be undergoing a thermal shutdown at the datacenter.

August 14, 2020, 20:20 AM EST: Network problems to niagara/mist. We are investigating.


{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[SOSCIP_GPU | SOSCIP GPU cluster]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://support.scinet.utoronto.ca/education/browse.php SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH Tunneling]]
* [[SSH#Two-Factor_authentication|Two-Factor Authentication]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2020-08-15T00:59:54Z

Dgruner: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down|Niagara|Niagara_Quickstart}}
|{{Down|HPSS|HPSS}}
|{{Down|Mist|Mist}}
|{{Down|Teach|Teach}}
|-
|{{Down|Jupyter Hub|Jupyter_Hub}}
|{{Down|Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Down|File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Down|Burst Buffer|Burst_Buffer}}
|-
|{{Down|Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Down|External Network|Niagara_Quickstart#Logging_in}}
|{{Down|Globus|Globus}}
|}


August 14, 2020, 20:37 AM EST: We seem to be undergoing a thermal shutdown at the datacenter.

August 14, 2020, 20:20 AM EST: Network problems to niagara/mist. We are investigating.

August 13, 2020, 10:40 AM EST: Network is fixed, scheduler and other services are back.

August 13, 2020, 8:20 AM EST: We had an IB switch failure, which is affecting a subset of nodes, including the scheduler nodes.

August 10, 2020, 7:30 PM EST: Scheduler fully operational again.

August 10, 2020, 3:00 PM EST: Scheduler partially functional: jobs can be submitted and are running.

August 10, 2020, 2:00 PM EST: Scheduler is temporarily inoperational.

August 7, 2020, 9:15 PM EST: Network is fixed, scheduler and other services are coming back.

August 7, 2020, 8:20 PM EST: Disruption of part of the network in the data centre. Causes issue with the scheduler, the mist login node, and possibly others. We are investigating.


{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[SOSCIP_GPU | SOSCIP GPU cluster]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://support.scinet.utoronto.ca/education/browse.php SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH Tunneling]]
* [[SSH#Two-Factor_authentication|Two-Factor Authentication]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2020-03-20T17:16:52Z

Dgruner: /* System Status */

MATLAB

2020-03-11T17:49:43Z

Dgruner: /* Using a MATLAB stand-alone executable on Niagara */

We often get questions about running MATLAB on Niagara. With a few exceptions for compilers and debuggers, SciNet does not purchase licenses for commercial software. As such, SciNet does not have a license for MATLAB, nor will it in the future. If users wish to run MATLAB they must supply their own license, or explore alternative options. This page gives information about the options for getting your MATLAB code to run, in recommended order.

== Not using MATLAB ==

Users can attempt to run MATLAB code using the open-source program [[Octave]], accessible through the octave module. Though there are some differences between the two programs, Octave has been designed to interpret MATLAB code and can often be used in place of MATLAB. If your MATLAB code does not use some of the more-fancy MATLAB toolboxes, you may be able to get away with using Octave instead. Be sure to test your implementation in Octave thoroughly before committing to this option.

It is worth observing that, while convenient for prototyping and running on a single workstation, there are reasons to avoid using MATLAB for larger HPC/ARC projects. These include the prohibitive license cost for large-scale work, poor performance at scale, and portability issues. If you can switch to a license-free option, such as Python, it may be worth the effort.

== Using stand-alone MATLAB executables ==

=== Creating a MATLAB stand-alone executable ===

If MATLAB must be used, you may be able to compile your MATLAB code into a stand-alone executable, and run this on a Niagara compute node. The version of MATLAB being used will require a compiler license, and the compilation must be done on a Linux machine (not Niagara).

=== Using a MATLAB stand-alone executable on Niagara ===

Once the compilation is done, the executable can be copied to SciNet, and run using the MATLAB Compiler Runtime (MCR), which can be accessed using the MCR module. The MCR used must be the same version of MATLAB as the compiler. If the version of MCR that you need is not listed among the MCR module versions, contact us and we will install the version which you require.

Here is an example script which uses the MCR. Note that the "run_myscript.sh" script is produced by the MATLAB compiler, together with the "myscript" executable (assuming you were working on the myscript.m MATLAB code):

<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=40
#SBATCH --time=1:00:00
#SBATCH --job-name test_matlab
#SBATCH --output=matlab_output_%j.txt

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is the directory from which the job was submitted
cd $SLURM_SUBMIT_DIR

# load module
module load mcr/R2018a

# Directory for the MCR to use to write temporary files. Use whatever directory you wish.
mkdir -p $SCRATCH/temp
export MCR_CACHE_ROOT=$SCRATCH/temp

# EXECUTION COMMAND (note that the MATLAB script may require that LD_LIBRARY_PATH be added
# to the script arguments). Note that, if the calculations are serial, you must bundle 40 such
# calculations together for production runs!
./run_myscript.sh $MATLAB:$LD_LIBRARY_PATH
</source>

=== Avaiable MATLAB runtime versions ===

Below is a list of the available MATLAB runtime versions on Niagara, with the required module command:

module load NiaEnv/2018a mcr/R2018a
module load NiaEnv/2019b mcr/R2019a
module load CCEnv nixpkgs/16.09 mcr/R2013a
module load CCEnv nixpkgs/16.09 mcr/R2013b
module load CCEnv nixpkgs/16.09 mcr/R2014a
module load CCEnv nixpkgs/16.09 mcr/R2014b
module load CCEnv nixpkgs/16.09 mcr/R2015a
module load CCEnv nixpkgs/16.09 mcr/R2015b
module load CCEnv nixpkgs/16.09 mcr/R2016a
module load CCEnv nixpkgs/16.09 mcr/R2016b
module load CCEnv nixpkgs/16.09 mcr/R2017a
module load CCEnv nixpkgs/16.09 mcr/R2017b

== Tunneling to a license server ==

If you have access to a non-SciNet MATLAB license server, and have installed MATLAB in your $HOME directory, you can [[SSH_Tunneling|setup your submission script]] to access the external license server. The following lines should be added to the beginning of your submission script, after the #SBATCH commands:

<pre>
PORT=XXX # port number of the license server
LICENSE_IP=AAA.BBB.CCC.DDD # IP address of the license server
ssh nia-gw -L${PORT}:${LICENSE_IP}:${PORT} -N &
</pre>

This last line will tunnel the port from the compute node back to the license server, through nia-gw. The port number and IP address of the licence server must be supplied by the system administrator of the license server.

== Using a different Consortium ==

Both [https://www.sharcnet.ca/my/software/show/54 Sharcnet] and [https://www.westgrid.ca/support/software/matlab Westgrid] have purchased different types of MATLAB licenses. Users can contact those consortia if they wish to attempt to run on those systems.

SSH Changes in May 2019

2019-05-30T22:23:02Z

Dgruner: /* Qu'est-ce qui a changé? */

=English=
== What Changed? ==

During the 29-30 May 2019 shutdown, we made the following ssh security improvements on Niagara:

# Disabled certain weak encryption algorithms.
# Disabled certain weak public key types.
# Regenerated Niagara's host keys.

== Updating your client's known host list ==

The first time you login to Niagara after the shutdown, you will probably see the following warning message:

<pre>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:SauX2nL+Yso9KBo2Ca6GH/V9cSFLFXwxOECGWXZ5pxc.
Please contact your system administrator.
Add correct host key in /home/username/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/username/.ssh/known_hosts:109
ED25519 host key for niagara.scinet.utoronto.ca has changed and you have requested strict checking.
Host key verification failed.
Killed by signal 1.
</pre>

This warning is displayed because the host keys on Niagara changed to increase the data centres security, and ssh clients remember old host keys to prevent [https://en.wikipedia.org/wiki/Man-in-the-middle_attack "man-in-the-middle" attacks].

You may also get a warning regarding "DNS spoofing", which is related to the same change.

If you are using MobaXTerm or Putty as your ssh client under Windows, the warning will appear in a pop-up window and will allow you to accept the new host key by clicking "Yes".

If you are using the command line ssh command on macOS, Linux, GitBash or Cygwin, you should tell your system to "forget" the old host keys, by running the following commands:

$ ssh-keygen -R niagara.scinet.utoronto.ca
$ ssh-keygen -R niagara.computecanada.ca
$ ssh-keygen -R 142.150.188.70

Afterwards, the next time you ssh to Niagara you'll be asked to confirm the new host keys, e.g.:

$ ssh niagara.scinet.utoronto.ca
The authenticity of host 'niagara.scinet.utoronto.ca (142.150.188.70)' can't be established.
ED25519 key fingerprint is SHA256:SauX2nL+Yso9KBo2Ca6GH/V9cSFLFXwxOECGWXZ5pxc.
ED25519 key fingerprint is MD5:b4:ae:76:a5:2b:37:8d:57:06:0e:9a:de:62:00:26:be.
Are you sure you want to continue connecting (yes/no)?

Make sure the fingerprints are correct! You'll either see the above ED25519 fingerprints, or the following RSA fingerprints:

RSA key fingerprint is SHA256:k6YEhYsI73M+NJIpZ8yF+wqWeuXS9avNs2s5QS/0VhU.
RSA key fingerprint is MD5:98:e7:7a:07:89:ef:3f:d8:68:3d:47:9c:6e:a6:71:5e.

If the fingerprints don't match, someone may be trying to hijack your connection.

WINSCP?

== Troubleshooting ==

=== I can't connect! ===

If you see one of the following error messages:

Unable to negotiate with 142.150.188.70 port 22: no matching cipher found.
Unable to negotiate with 142.150.188.70 port 22: no matching key exchange method found.
Unable to negotiate with 142.150.188.70 port 22: no matching mac found.

you need to upgrade your ssh client.

=== My SSH key no longer works ===

If you're being asked for a password, but were using SSH keys,
it's because 1024-bit DSA & RSA keys have been disabled.

You need to generate a new stronger key, see the [[SSH keys]] page for details.

=Les changement de SSH à mai 2019 (version française)=

==Qu'est-ce qui a changé?==

Pendant l'arrêt planifié des 29 et 30 mai 2019, nous avons mis en place
les mesures suivantes à Niagara pour améliorer sa securité:

# Certaines méthodes de chiffrement faibles ont été désactivées.
# Certains types de clé publique SSH ont été désactivés.
# Les clés d'hôte de Niagara ont été régénérées.

==Mettre à jour la liste de clés d'hôte de votre client SSH==

Lorsque vous vous reconnectez à Niagara pour la première fois apres
l'arrêt de maintenance, vous verrez probablement le message suivante:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:SauX2nL+Yso9KBo2Ca6GH/V9cSFLFXwxOECGWXZ5pxc.
Please contact your system administrator.
Add correct host key in /home/username/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/username/.ssh/known_hosts:109
ED25519 host key for niagara.scinet.utoronto.ca has changed and you have requested strict checking.
Host key verification failed.
Killed by signal 1.

Ce message d'avertissement s'affiche parce que les clés d'hôte de Niagara ont été régénérées pour améliorer le securité de le centre de données, mais le client SSH a stocké les clés d'hôte anciennes pour empêcher une [https://fr.wikipedia.org/wiki/Attaque_de_l%27homme_du_milieu « attaque de l'homme du milieu »].

C'est possible que vous verrez aussi un message d'avertissement sur « DNS spoofing ».

Si vous utilisez un des outils de connexion MobaXTerm et PuTTY, le message d'avertissment s'affiche dans une fenêtre contextuelle qui offre l'option d'accepter la nouvelle clé de hôte par appuyer sur le bouton approprié.

Si vous utilisez le command « ssh » dans une fenêtre de terminal sur macOS, Linux, GitBash ou Cygwin, vous devez dire à votre système d'oublier les anciennes clés de l'hôte avec les commandes suivantes:

$ ssh-keygen -R niagara.scinet.utoronto.ca
$ ssh-keygen -R niagara.computecanada.ca
$ ssh-keygen -R 142.150.188.70

Ensuite, lors de votre prochaine visite à Niagara, vous devrez confirmer les nouvelles clés d’hôte, par exemple:

$ ssh niagara.scinet.utoronto.ca
The authenticity of host 'niagara.scinet.utoronto.ca (142.150.188.70)' can't be established.
ED25519 key fingerprint is SHA256:SauX2nL+Yso9KBo2Ca6GH/V9cSFLFXwxOECGWXZ5pxc.
ED25519 key fingerprint is MD5:b4:ae:76:a5:2b:37:8d:57:06:0e:9a:de:62:00:26:be.
Are you sure you want to continue connecting (yes/no)?

Assurez-vous que les empreintes digitales (« fingerprints ») sont correctes! Vous verrez soit les empreintes ED25519 ci-dessus, soit les empreintes RSA suivantes:

RSA key fingerprint is SHA256:k6YEhYsI73M+NJIpZ8yF+wqWeuXS9avNs2s5QS/0VhU.
RSA key fingerprint is MD5:98:e7:7a:07:89:ef:3f:d8:68:3d:47:9c:6e:a6:71:5e.

Si les empreintes digitales ne correspondent pas, il est possible que quelqu'un tente de détourner votre connexion.

==Dépannage==

===Je n'arrive pas à me connecter!===

Si vous voyez l'un des messages d'erreur suivants:

Unable to negotiate with 142.150.188.70 port 22: no matching cipher found.
Unable to negotiate with 142.150.188.70 port 22: no matching key exchange method found.
Unable to negotiate with 142.150.188.70 port 22: no matching mac found.

vous devez mettre à jour le client ssh.

===Ma clé SSH ne fonctionne plus===

Si un mot de passe vous est demandé alors que vous utilisiez auparavant des clés SSH, c'est probablement parce que les clés DSA et RSA 1024 bits ont été désactivées.

Vous devez générer une nouvelle clé plus puissante, voir la page [[SSH keys]] pour plus de détails.

Main Page

2019-04-10T02:24:39Z

Dgruner: /* System Status */

NAMD

2019-03-31T16:16:24Z

Dgruner: /* NAMD v2.13 */

====NAMD v2.13====

This is the NAMD version 2.13 Scalable Molecular Dynamics package.
NAMD_2.13_Linux-x86_64-ibverbs-smp.tar.gz binary tarball from TCBG.

It was built directly on top of ibverbs, plus smp, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

module load namd/2.13

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=2
P=$(($SLURM_NTASKS * 2))

charmrun ++verbose +p $P ++ppn $PPN ++nodelist $NODELIST $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

====NAMD v2.12====

This is the NAMD version 2.12 Scalable Molecular Dynamics package.

It was built directly on top of ibverbs, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

# Note that the module will likely be taken out of experimental mode at some point.
module load namd/.experimental-2.12-ibverbs-smp

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=4
P=$(($SLURM_NTASKS * 2))

charmrun ++verbose +p $P ++ppn $PPN ++nodelist $NODELIST $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

NAMD

2019-03-31T14:42:01Z

Dgruner: /* NAMD v2.13 */

====NAMD v2.13====

This is the NAMD version 2.13 Scalable Molecular Dynamics package.
NAMD_2.13_Linux-x86_64-ibverbs-smp.tar.gz binary tarball from TCBG.

It was built directly on top of ibverbs, plus smp, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

# The intel and openmpi modules are needed in order to have mpiexec in the path, to use it to launch the processes
module load intel
module load openmpi
module load namd/2.13

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=2
P=$(($SLURM_NTASKS * 2))

charmrun +p<procs> ++ppn $PPN ++mpiexec $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

====NAMD v2.12====

This is the NAMD version 2.12 Scalable Molecular Dynamics package.

It was built directly on top of ibverbs, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

# Note that the module will likely be taken out of experimental mode at some point.
module load namd/.experimental-2.12-ibverbs-smp

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=4
P=$(($SLURM_NTASKS * 2))

charmrun ++verbose +p $P ++ppn $PPN ++nodelist $NODELIST $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

NAMD

2019-03-31T14:33:40Z

Dgruner: /* NAMD v2.13 */

====NAMD v2.13====

This is the NAMD version 2.13 Scalable Molecular Dynamics package.
NAMD_2.13_Linux-x86_64-ibverbs-smp.tar.gz binary tarball from TCBG.

It was built directly on top of ibverbs, plus smp, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

# The intel and openmpi modules are needed in order to have mpiexec in the path, to use it to launch the processes
module load intel
module load openmpi
module load namd/2.13

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=4
P=$(($SLURM_NTASKS * 2))

charmrun +p<procs> ++ppn $PPN ++mpiexec $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

====NAMD v2.12====

This is the NAMD version 2.12 Scalable Molecular Dynamics package.

It was built directly on top of ibverbs, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

# Note that the module will likely be taken out of experimental mode at some point.
module load namd/.experimental-2.12-ibverbs-smp

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=4
P=$(($SLURM_NTASKS * 2))

charmrun ++verbose +p $P ++ppn $PPN ++nodelist $NODELIST $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

NAMD

2019-03-30T23:49:05Z

Dgruner: /* NAMD v2.13 */

====NAMD v2.13====

This is the NAMD version 2.13 Scalable Molecular Dynamics package.
NAMD_2.13_Linux-x86_64-ibverbs-smp.tar.gz binary tarball from TCBG.

It was built directly on top of ibverbs, plus smp, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

# The intel and openmpi modules are needed in order to have mpiexec in the path, to use it to launch the processes
module load intel
module load openmpi
module load namd/2.13

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=4
P=$(($SLURM_NTASKS * 2))

charmrun +p<procs> ++mpiexec $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

====NAMD v2.12====

This is the NAMD version 2.12 Scalable Molecular Dynamics package.

It was built directly on top of ibverbs, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

# Note that the module will likely be taken out of experimental mode at some point.
module load namd/.experimental-2.12-ibverbs-smp

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=4
P=$(($SLURM_NTASKS * 2))

charmrun ++verbose +p $P ++ppn $PPN ++nodelist $NODELIST $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

NAMD

2019-03-30T23:43:12Z

Dgruner: /* NAMD v2.12 */

====NAMD v2.13====

This is the NAMD version 2.13 Scalable Molecular Dynamics package.

It was built directly on top of ibverbs, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

# The intel and openmpi modules are needed in order to have mpiexec in the path, to use it to launch the processes
module load intel
module load openmpi
module load namd/2.13

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=4
P=$(($SLURM_NTASKS * 2))

charmrun +p<procs> ++mpiexec $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

====NAMD v2.12====

This is the NAMD version 2.12 Scalable Molecular Dynamics package.

It was built directly on top of ibverbs, so it will run on InfiniBand nodes. Here is a sample run script:

<source lang="bash">
#!/bin/bash
#
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --time=00:15:00
#SBATCH --job-name namdtest

# Note that the module will likely be taken out of experimental mode at some point.
module load namd/.experimental-2.12-ibverbs-smp

# DIRECTORY TO RUN - $SLURM_SUBMIT_DIR is directory job was submitted from
cd $SLURM_SUBMIT_DIR

# Generate NAMD nodelist
for n in `echo $SLURM_NODELIST | scontrol show hostnames`; do
echo "host $n" >> nodelist.$SLURM_JOBID
done

NODELIST=nodelist.$SLURM_JOBID
cat $NODELIST

# Calculate total processes (P) and procs per node (PPN)
PPN=4
P=$(($SLURM_NTASKS * 2))

charmrun ++verbose +p $P ++ppn $PPN ++nodelist $NODELIST $SCINET_NAMD_ROOT/bin/namd2 input.namd

# Cleaning
rm $NODELIST
</source>

Full documentation for NAMD is available on their website: http://www.ks.uiuc.edu/Research/namd/

FAQ

2018-07-30T14:58:46Z

Dgruner: /* How do priorities work/why did that job jump ahead of mine in the queue? */

__TOC__

==The Basics==
===Whom do I contact for support?===

Whom do I contact if I have problems or questions about how to use the SciNet systems?

'''Answer:'''

E-mail [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]

In your email, please include the following information:

* your username on SciNet
* the cluster that your question pertains to (NIA, BGQ, GPU, ...; SciNet is not a cluster!),
* any relevant error messages
* the commands you typed before the errors occurred
* the path to your code (if applicable)
* the location of the job scripts (if applicable)
* the directory from which it was submitted (if applicable)
* a description of what it is supposed to do (if applicable)
* if your problem is about connecting to SciNet, the type of computer you are connecting from.

Note that your password should never, never, never be to sent to us, even if your question is about your account.

Avoid sending email only to specific individuals at SciNet. Your chances of a quick reply increase significantly if you email our team! (support@scinet.utoronto.ca)

===What does ''code scaling'' mean?===

'''Answer:'''

Please see [[Introduction_To_Performance#Parallel_Speedup|A Performance Primer]]

===What do you mean by ''throughput''?===

'''Answer:'''

Please see [[Introduction_To_Performance#Throughput|A Performance Primer]].

Here is a simple example:

Suppose you need to do 10 computations. Say each of these runs for
1 day on 8 cores, but they take "only" 18 hours on 16 cores. What is the
fastest way to get all 10 computations done - as 8-core jobs or as
16-core jobs? Let us assume you have 2 nodes at your disposal.
The answer, after some simple arithmetic, is that running your 10
jobs as 8-core jobs will take 5 days, whereas if you ran them
as 16-core jobs it would take 7.5 days. Take your own conclusions...

===I changed my .bashrc/.bash_profile and now nothing works===

The default startup scripts provided by SciNet, and guidelines for them, can be found [[Important_.bashrc_guidelines|here]]. Certain things - like sourcing <tt>/etc/profile</tt>
and <tt>/etc/bashrc</tt> are ''required'' for various SciNet routines to work!

If the situation is so bad that you cannot even log in, please send email [mailto:support@scinet.utoronto.ca support].

===Could I have my login shell changed to (t)csh?===

The login shell used on our systems is bash. While the tcsh is available on the GPC and the TCS, we do not support it as the default login shell at present. So "chsh" will not work, but you can always run tcsh interactively. Also, csh scripts will be executed correctly provided that they have the correct "shebang" <tt>#!/bin/tcsh</tt> at the top.

===How can I run Matlab / IDL / Gaussian / my favourite commercial software at SciNet?===

'''Answer:'''

Because SciNet serves such a disparate group of user communities, there is just no way we can buy licenses for everyone's commercial package. The only commercial software we have purchased is that which in principle can benefit everyone -- fast compilers and math libraries (Intel's on GPC, and IBM's on TCS).

If your research group requires a commercial package that you already have or are willing to buy licenses for, contact us at [mailto:support@scinet.utoronto.ca support@scinet] and we can work together to find out if it is feasible to implement the packages licensing arrangement on the SciNet clusters, and if so, what is the the best way to do it.

Note that it is important that you contact us before installing commercially licensed software on SciNet machines, even if you have a way to do it in your own directory without requiring sysadmin intervention. It puts us in a very awkward position if someone is found to be running unlicensed or invalidly licensed software on our systems, so we need to be aware of what is being installed where.

===Do you have a recommended ssh program that will allow scinet access from Windows machines?===

'''Answer:'''

The [[Ssh#SSH_for_Windows_Users | SSH for Windows users]] programs we recommend are:

* [http://mobaxterm.mobatek.net/en/ MobaXterm] is a tabbed ssh client with some Cygwin tools, including ssh and X, all wrapped up into one executable.
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] - this is a terminal for windows that connects via ssh. It is a quick install and will get you up and running quickly. '''WARNING:''' Make sure you download putty from the official website, because there are "trojanized" versions of putty around that will send your login information to a site in Russia (as reported [http://blogs.cisco.com/security/trojanized-putty-software here]). To set up your passphrase protected ssh key with putty, see [http://the.earth.li/~sgtatham/putty/0.61/htmldoc/Chapter8.html#pubkey here].
* [http://www.cygwin.com/ CygWin] - this is a whole linux-like environment for windows, which also includes an X window server so that you can display remote windows on your desktop. Make sure you include the openssh and X window system in the installation for full functionality. This is recommended if you will be doing a lot of work on Linux machines, as it makes a very similar environment available on your computer. To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.
 To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.

===My ssh key does not work! WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! ===

'''Answer:'''

[[Ssh_keys#Testing_Your_Key | Testing Your Key]]

* If this doesn't work, you should be able to login using your password, and investigate the problem. For example, if during a login session you get an message similar to the one below, just follow the instruction and delete the offending key on line 3 (you can use vi to jump to that line with ESC plus : plus 3). That only means that you may have logged in from your home computer to SciNet in the past, and that key is obsolete.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle
attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
53:f9:60:71:a8:0b:5d:74:83:52:**fe:ea:1a:9e:cc:d3.
Please contact your system administrator.
Add correct host key in /home/<user>/.ssh/known_hosts to get rid of
this message.
Offending key in /home/<user>/.ssh/known_hosts:3
RSA host key for login.scinet.utoronto.ca
<http://login.scinet.utoronto.ca <http://login.scinet.utoronto.ca>> has
changed and you have requested
</pre>

* If you get the message below you may need to logout of your gnome session and log back in since the ssh-agent needs to be
restarted with the new passphrase ssh key.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

Agent admitted failure to sign using the key.
</pre>

===Can't get graphics: "Can't open display/DISPLAY is not set"===

To use graphics on SciNet machines and have it displayed on your machine, you need to have a X server running on your computer (an X server is the standard way graphics is done on linux). One an X server is running, you can log in with the "-Y" option to ssh ("-X" sometimes also works).

How to get an X server running on your computer, depends on the operating system. On linux machines with a graphical interface, X will already be running. On windows, the easiest solution is using MobaXterm, which comes with an X server (alternatives, such as cygwin with the x11 server installed, or running putty+Xming, can also work, but are a bit more work to set up. For Macs, you will need to install Xquartz.

===Remote graphics stops working after a while: "Can't open display"===

If you still cannot get graphics, or it works only for a while and then suddenly it "can't open display localhost:....", your X11 graphics connection may have timed out (Macs seem to be particularly prone to this). You'll have to tell your own computer not to allow, and not to timeout the X11 graphics connection.

The following should fix it. The ssh configuration settings are in a file called /etc/ssh/ssh_config (or /etc/ssh_config in older OS X versions, or $HOME/.ssh/config for specific users). In the config file, find (or create) the section "Host *" (meaning all hosts) and add the following lines:

Host *
ServerAliveInterval 60
ServerAliveCountMax 3
ForwardX11 yes
ForwardX11Trusted yes
ForwardX11Timeout 596h

(The <tt>Host *</tt> is only needed if there was no Host section yet to append these settings to.)

If this does not resolve it, try it again with "ssh -vvv -Y ....". The "-vvv" spews out a lot of diagnostic messages. Look for anything resembling a timeout, and let us know (support AT scinet DOT utoronto DOT ca).

===Can't forward X: "Warning: No xauth data; using fake authentication data", or "X11 connection rejected because of wrong authentication."===

I used to be able to forward X11 windows from SciNet to my home machine, but now I'm getting these messages; what's wrong?

'''Answer:'''

This very likely means that ssh/xauth can't update your ${HOME}/.Xauthority file.

The simplest pssible reason for this is that you've filled your 10GB /home quota and so can't write anything to your home directory. Use

<pre>
$ module load extras
$ diskUsage
</pre>

to check to see how close you are to your disk usage on ${HOME}.

Alternately, this could mean your .Xauthority file has become broken/corrupted/confused some how, in which case you can delete that file, and when you next log in you'll get a similar warning message involving creating .Xauthority, but things should work.

===I have a CCDB account, but I can't login to SciNet. How can I get a SciNet account?===

'''Answer:'''

You must extend your CCDB application process to also get a SciNet account:

https://wiki.scinet.utoronto.ca/wiki/index.php/Application_Process

https://www.scinethpc.ca/getting-a-scinet-account/

===How can I reset the password for my Compute Canada account?===

'''Answer:'''

You can reset your password for your Compute Canada account here:

https://ccdb.computecanada.ca/security/forgot

===How can I change or reset the password for my SciNet account?===

'''Answer:'''

To reset your password at SciNet please go to [https://portal.scinet.utoronto.ca/password_resets Password reset page].

If you know your old password and want to change it, that can be done here after logging in on the portal:

https://portal.scinet.utoronto.ca

===Why am I getting the error "Permission denied (publickey,gssapi-with-mic,password)"?===

This error can pop up in a variety of situations: when trying to log in, or when after a job has finished, when the error and output files fail to be copied (there are other possible reasons for this failure as well -- see [[FAQ#My_GPC_job_died.2C_telling_me_.60Copy_Stageout_Files_Failed.27|My GPC job died, telling me:Copy Stageout Files Failed]]).
In most cases, the "Permission denioed" error is caused by incorrect permission of the (hidden) .ssh directory. Ssh is used for logging in as well as for the copying of the standard error and output files after a job.

For security reasons,
the directory .ssh should only be writable and readable to you, but yours
has read permission for everybody, and thus it fails. You can change
this by
<pre>
chmod 700 ~/.ssh
</pre>
And to be sure, also do
<pre>
chmod 600 ~/.ssh/id_rsa ~/authorized_keys
</pre>

===ERROR:102: Tcl command execution failed? when loading modules ===
Modules sometimes require other modules to be loaded first.
Module will let you know if you didn’t.
For example:
<pre>
$ module purge
$ module load python
python/2.6.2(11):ERROR:151: Module ’python/2.6.2’ depends on one of the module(s) ’gcc/4.4.0’
python/2.6.2(11):ERROR:102: Tcl command execution failed: prereq gcc/4.4.0
$ gpc-f103n084-$ module load gcc python
$
</pre>

=== How do I compute the core-years usage of my code? ===

The "core-years" quantity is a way to account for the time your code runs, by considering the total number of cores and time used, accounting for the total number of hours in a year.
For instance if your code uses ''HH'' hours, in ''NN'' nodes, where each node has ''CC'' cores, then "core-years" can be computed as follow:

''HH*(NN*CC)/(365*24)''

If you have several independent instances (batches) running on different nodes, with ''BB'' number of batches and each batch during ''HH'' hours, then your core-years usage can be computed as,

''BB*HH*(NN*CC)/(365*24)''

As a general rule, in our GPC system, each node has only 8 cores, so ''CC'' will be always 8.

==Compiling your Code==

===How can I get g77 to work?===

The fortran 77 compilers on the GPC are ifort and gfortran. We have dropped support for g77. This has been a conscious decision. g77 (and the associated library libg2c) were completely replaced six years ago (Apr 2005) by the gcc 4.x branch, and haven't undergone any updates at all, even bug fixes, for over five years.
If we would install g77 and libg2c, we would have to deal with the inevitable confusion caused when users accidentally link against the old, broken, wrong versions of the gcc libraries instead of the correct current versions.

If your code for some reason specifically requires five-plus-year-old libraries, availability, compatibility, and unfixed-known-bug problems are only going to get worse for you over time, and this might be as good an opportunity as any to address those issues.

''A note on porting to gfortran or ifort:''

While gfortran and ifort are rather compatible with g77, one
important difference is that by default, gfortran does not preserve
local variables between function calls, while g77 does. Preserved
local variables are for instance often used in implementations of quasi-random number
generators. Proper fortran requires to declare such variables as SAVE
but not all old code does this.
Luckily, you can change gfortran's default behavior with the flag
<tt>-fno-automatic</tt>. For ifort, the corresponding flag is <tt>-noautomatic</tt>.

===Where is libg2c.so?===

libg2c.so is part of the g77 compiler, for which we dropped support. See [[#How can I get g77 to work on the GPC?]] for our reasons.

===Autoparallelization does not work!===

I compiled my code with the <tt>-qsmp=omp,auto</tt> option, and then I specified that it should be run with 64 threads - with
export OMP_NUM_THREADS=64

However, when I check the load using <tt>llq1 -n</tt>, it shows a load on the node of 1.37. Why?

'''Answer:'''

Using the autoparallelization will only get you so far. In fact, it usually does not do too much. What is helpful is to run the compiler with the <tt>-qreport</tt> option, and then read the output listing carefully to see where the compiler thought it could parallelize, where it could not, and the reasons for this. Then you can go back to your code and carefully try to address each of the issues brought up by the compiler.
We ''emphasize'' that this is just a rough first guide, and that the compilers are still not magical! For more sophisticated approaches to parallelizing your code, email us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] to set up an appointment with one
of our technical analysts.

===How do I link against the Intel Math Kernel Library?===

If you need to link to the Intel Math Kernal Library (MKL) with the intel compilers, just add the <pre>-mkl</pre> flag. There are in fact three flavours: <tt>-mkl=sequential</tt>, <tt>-mkl=parallel</tt> and <tt>-mkl=cluster</tt>, for the serial version, the threaded version and the mpi version, respectively. (Note: The cluster version is available only when using the intelmpi module and mpi compilation wrappers.)

If you need to link in the Intel Math Kernel Library (MKL) libraries to gcc/gfortran/c++, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code.

'''''Note that this give the link line for the command line. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.'''''

'''''Note too that, unless the integer arguments you will be passing to the MKL libraries are actually 64-bit integers, rather than the normal int or INTEGER types, you want to specify 32-bit integers (lp64) .'''''

===Can the compilers on the login nodes be disabled to prevent accidentally using them?===

'''Answer:'''

You can accomplish this by modifying your .bashrc to not load the compiler modules. See [[Important .bashrc guidelines]].

==="relocation truncated to fit: R_X86_64_PC32": Huh?===

What does this mean, and why can't I compile this code?

'''Answer:'''

Welcome to the joys of the x86 architecture! You're probably having trouble building arrays larger than 2GB, individually or together. Generally, you have to try to use the medium or large x86 `memory model'. For the intel compilers, this is specified with the compile options

-mcmodel=medium -shared-intel

==="feupdateenv is not implemented and will always fail"===

How do I get rid of this and what does it mean?

'''Answer:'''
First note that, as ominous as it sounds, this is really just a warning, and has to do with the intel math library. You can ignore it (unless you really are trying to manually change the exception handlers for floating point exceptions such as divide by zero), or take the safe road and get rid off it by linking with the intel math functions library:<pre>-limf</pre>See also [[#How do I link against the Intel Math Kernel Library?]]

===Cannot find rdmacm library when compiling on GPC===

I get the following error building my code on GPC: "<tt>ld: cannot find -lrdmacm</tt>". Where can I find this library?

'''Answer:'''

This library is part of the MPI libraries; if your compiler is having problems picking it up, it probably means you are mistakenly trying to compile on the login nodes (scinet01..scinet04). The login nodes aren't part of the GPC; they are for logging into the data centre only. From there you must go to the GPC or TCS development nodes to do any real work.

=== Why do I get this error when I try to compile: "icpc: error #10001: could not find directory in which /usr/bin/g++41 resides" ?===

You are trying to compile on the login nodes. As described in the wiki ( https://support.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Login ), or in the users guide you would have received with your account, Scinet supports two main clusters, with very different architectures. Compilation must be done on the development nodes of the appropriate cluster (in this case, gpc01-04). Thus, log into gpc01, gpc02, gpc03, or gpc04, and compile from there.

==Testing your Code==

=== Can I run a something for a short time on the development nodes? ===

I am in the process of playing around with the mpi calls in my code to get it to work. I do a lot of tests and each of them takes a couple of seconds only. Can I do this on the development nodes?

'''Answer:'''

Yes, as long as it's very brief (a few minutes). People use the development nodes
for their work, and you don't want to bog it down for people, and testing a real
code can chew up a lot more resources than compiling, etc. The procedures differ
depending on what machine you're using.

==== TCS ====

On the TCS you can run small MPI jobs on the tcs02 node, which is meant for
development use. But even for this test run on one node, you'll need a host file --
a list of hosts (in this case, all tcs-f11n06, which is the `real' name of tcs02)
that the job will run on. Create a file called `hostfile' containing the following:

tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06

for a 4-task run. When you invoke "poe" or "mpirun", there are runtime
arguments that you specify pointing to this file. You can also specify it
in an environment variable MP_HOSTFILE, so, if your file is in your /scratch directory, say
${SCRATCH}/hostfile, then you would do

<pre>
export MP_HOSTFILE=${SCRATCH}/hostfile
</pre>

in your shell. You will also need to create a <tt>.rhosts</tt> file in your
home director, again listing <tt>tcs-f11n06</tt> so that <tt>poe</tt>
can start jobs. After that you can simply run your program. You can use
mpiexec:

<pre>
mpiexec -n 4 my_test_program
</pre>

adding <tt> -hostfile /path/to/my/hostfile</tt> if you did not set the environment
variable above. Alternatively, you can run it with the poe command (do a "man poe" for details), or even by
just directly running it. In this case the number of MPI processes will by default
be the number of entries in your hostfile.

==== GPC ====

On the GPC one can run short test jobs on the GPC [[GPC_Quickstart#Compile.2FDevel_Nodes | development nodes ]]<tt>gpc01</tt>..<tt>gpc04</tt>;
if they are single-node jobs (which they should be) they don't need a hostfile. Even better, though, is to request an [[ Moab#Interactive | interactive ]] job and run the tests either in regular batch queue or using a short high availability [[ Moab#debug | debug ]] queue that is reserved for this purpose.

=== How do I run a longer (but still shorter than an hour) test job quickly ? ===

'''Answer'''

On the GPC there is a high turnover short queue called [[ Moab#debug | debug ]] that is designed for
this purpose. You can use it by adding
<pre>
#PBS -q debug
</pre>
to your submission script.

==Submitting your jobs==

===Error Submitting My Job: qsub: Bad UID for job execution MSG=ruserok failed ===

I write up a submission script as in the examples, but when I attempt to submit the job, I get the above error. What's wrong?

'''Answer:'''

This error will occur if you try to submit a job from the login nodes. The login nodes are the gateway to all of SciNet's systems (GPC, TCS, P7, ARC), which have different hardware and queueing systems. To submit a job, you must log into a development node for the particular cluster you are submitting to and submit from there.

=== How do I charge jobs to my RAC allocation? ===

'''Answer:'''

Please see the [[Moab#Accounting|accounting section of Moab page]].

===How can I automatically resubmit a job?===

Commonly you may have a job that you know will take longer to run than what is
permissible in the queue. As long as your program contains [[Checkpoints|checkpoint]] or
restart capability, you can have one job automatically submit the next. In
the following example it is assumed that the program finishes before
the 48 hour limit and then resubmits itself by logging into one
of the development nodes.

<source lang="bash">
#!/bin/bash
# MOAB/Torque example submission script for auto resubmission
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=48:00:00
#PBS -N my_job

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# YOUR CODE HERE
./run_my_code

# RESUBMIT 10 TIMES HERE
num=$NUM
if [ "$num" -lt 10 ]; then
num=$(($num+1))
ssh gpc01 "cd $PBS_O_WORKDIR; qsub ./script_name.sh -v NUM=$num";
fi
</source>

<pre>
qsub script_name.sh -v NUM=0
</pre>

You can alternatively use [[ Moab#Job_Dependencies | Job dependencies ]] through the queuing system which will not start one job until another job has completed.

If your job can't be made to automatically stop before the 48 hour queue window, but it does write out checkpoints, you can use the timeout command to stop the program while you still have time to resubmit; for instance

<source lang="bash">
timeout 2850m ./run_my_code argument1 argument2
</source>

will run the program for 47.5 hours (2850 minutes), and then send it SIGTERM to exit the program.

===How can I pass in arguments to my submission script?===

If you wish to make your scripts more generic you can use qsub's ability
to pass in environment variables to pass in arguments to your script.
The following example shows a case where an input and an output
file are passed in on the qsub line. Multiple variables can be
passed in using the qsub "-v" option and comma delimited.

<source lang="bash">
#!/bin/bash
# MOAB/Torque example of passing in arguments
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=48:00:00
#PBS -N my_job

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# YOUR CODE HERE
./run_my_code -f $INFILE -o $OUTFILE
</source>

<pre>
qsub script_name.sh -v INFILE=input.txt,OUTFILE=outfile.txt
</pre>

===I submit my GPC job, and I get an email saying it was rejected===
'''Answer:'''

This happens because the job you've submitted breaks one of the rules of the queues and is rejected. An email
is sent with the JOBID, JOBNAME, and the reason it was rejected. The following is an example where a job
requests more than 48 hours and was rejected.

<pre>
PBS Job Id: 3462493.gpc-sched
Job Name: STDIN
job deleted
Job deleted at request of root@gpc-sched
MOAB_INFO: job was rejected - job violates class configuration 'wclimit too high for class 'batch_ib' (345600 > 172800)'
</pre>

Jobs on the TCS or GPC may only run for 48 hours at a time; this restriction greatly increases responsiveness of the queue and queue throughput for all our users. If your computation requires longer than that, as many do, you will have to [[ Checkpoints | checkpoint ]] your job and restart it after each 48-hour queue window. You can manually re-submit jobs, or if you can have your job cleanly exit before the 48 hour window, there are ways to [[ FAQ#How_can_I_automatically_resubmit_a_job.3F | automatically resubmit jobs ]].

Other rejections return a more cryptic error saying "job violates class configuration" such as follows:
<pre>
PBS Job Id: 3462409.gpc-sched
Job Name: STDIN
job deleted
Job deleted at request of root@gpc-sched
MOAB_INFO: job was rejected - job violates class configuration 'user required by class 'batch''
</pre>

The most common problems that result in this error are:

* '''Incorrect number of processors per node''': Jobs on the GPC are scheduled per-node not per-core and since each node has 8 processor cores (ppn=8) the smallest job allowed is one node with 8 cores (nodes=1:ppn=8). For serial jobs users must bundle or batch them together in groups of 8. See [[ FAQ#How_do_I_run_serial_jobs_on_GPC.3F | How do I run serial jobs on GPC? ]]
* '''No number of nodes specified''': Jobs submitted to the main queue must request a specific number of nodes, either in the submission script (with a line like <tt>#PBS -l nodes=2:ppn=8</tt>) or on the command line (eg, <tt>qsub -l nodes=2:ppn=8,walltime=5:00:00 script.pbs</tt>). Note that for the debug queue, you can get away without specifying a number of nodes and a default of one will be assigned; for both technical and policy reasons, we do not enforce such a default for the main ("batch") queue.
* '''There is a 15 minute walltime minimum''' on all queues except debug and if you set your walltime less than this, it will be rejected.

=== When submitting your job, fails saying: "script is written in DOS/Windows text format" ===

'''Answer:'''

Very likely you have written your script in a windows machine, so to fix this you just need to change the format of you submission script to unix from Windows/DOS.
Use the command below for all your script files:

<pre>
dos2unix <pbs-script-file>
</pre>

where <pbs-script-file> has to be substituted by the name of your script file.

==Running your jobs==

===My job can't write to /home===

My code works fine when I test on the development nodes, but when I submit a job, or even run interactively in the development queue on GPC, it fails. What's wrong?

'''Answer:'''

As [[Data_Management#Home_Disk_Space | discussed]] [https://support.scinet.utoronto.ca/wiki/images/5/54/SciNet_Tutorial.pdf elsewhere], <tt>/home</tt> is mounted read-only on the compute nodes; you can only write to <tt>/home</tt> from the login nodes and devel nodes. (The [[GPC_Quickstart#128Glargemem | largemem nodes]] on GPC, in this respect, are more like devel nodes than compute nodes). In general, to run jobs you can read from <tt>/home</tt> but you'll have to write to <tt>/scratch</tt> (or, if you were allocated space through the RAC process, on <tt>/project</tt>). More information on SciNet filesytems can be found on our [[Data_Management | Data Management]] page.

===OpenMP on the TCS===

How do I run an OpenMP job on the TCS?

'''Answer:'''

Please look at the [[TCS_Quickstart#Submission_Script_for_an_OpenMP_Job | TCS Quickstart ]] page.

===Can I can use hybrid codes consisting of MPI and openMP on the GPC?===

'''Answer:'''

Yes. Please look at the [[GPC_Quickstart#Hybrid_MPI.2FOpenMP_jobs | GPC Quickstart ]] page.

===How do I run serial jobs on GPC?===

'''Answer''':

So it should be said first that SciNet is a parallel computing resource,
and our priority will always be parallel jobs. Having said that, if
you can make efficient use of the resources using serial jobs and get
good science done, that's good too, and we're happy to help you.

The GPC nodes each have 8 processing cores, and making efficient use of these
nodes means using all eight cores. As a result, we'd like to have the
users take up whole nodes (eg, run multiples of 8 jobs) at a time.

It depends on the nature of your job what the best strategy is. Several approaches are presented on the [[User_Serial|serial run wiki page]].

===Why can't I request only a single cpu for my job on GPC?===

'''Answer''':

On GPC, computers are allocated by the node - that is, in chunks of 8 processors. If you want to run a job that requires only one processor, you need to bundle the jobs into groups of 8, so as to not be wasting the other 7 for 48 hours. See [[User_Serial|serial run wiki page]].

===How do I run serial jobs on TCS?===

'''Answer''': You don't.

===But in the queue I found a user who is running jobs on GPC, each of which is using only one processor, so why can't I?===

'''Answer''':

The pradat* and atlaspt* jobs, amongst others, are jobs of the ATLAS high energy physics project. That they are reported as single cpu jobs is an artifact of the moab scheduler. They are in fact being automatically bundled into 8-job bundles but have to run individually to be compatible with their international grid-based systems.

===How do I use the ramdisk on GPC?===

To use the ramdisk, create and read to / write from files in /dev/shm/.. just as one would to (eg) ${SCRATCH}. Only the amount of RAM needed to store the files will be taken up by the temporary file system; thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.

It is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.

''More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].''

=== How can I run a job longer than 48 hours? ===

'''Answer:'''

The SciNet queues have a queue limit of 48 hours. This is pretty typical for systems of its size in Canada and elsewhere, and larger systems commonly have shorter limits. The limits are there to ensure that every user gets a fair share of the system (so that no one user ties up lots of nodes for a long time), and for safety (so that if one memory board in one node fails in the middle of a very long job, you haven't lost a months' worth of work).

Since many of us have simulations that require more than that much time, most widely-used scientific applications have "checkpoint-restart" functionality, where every so often the complete state of the calculation is stored as a checkpoint file, and one can restart a simulation from one of these. In fact, these restart files tend to be quite useful for a number of purposes.

If your job will take longer, you will have to submit your job in multiple parts, restarting from a checkpoint each time. In this way, one can run a simulation much longer than the queue limit. In fact, one can even write job scripts which automatically re-submit themselves until a run is completed, using [[FAQ#How_can_I_automatically_resubmit_a_job.3F | automatic resubmission. ]]

=== Why did showstart say it would take 3 hours for my job to start before, and now it says my job will start in 10 hours? ===

'''Answer:'''

Please look at the [[FAQ#How_do_priorities_work.2Fwhy_did_that_job_jump_ahead_of_mine_in_the_queue.3F | How do priorities work/why did that job jump ahead of mine in the queue? ]] page.

===How do priorities work/why did that job jump ahead of mine in the queue?===

'''Answer:'''

The [[Moab | queueing system]] used on SciNet machines is a [http://en.wikipedia.org/wiki/Priority_queue Priority Queue]. Jobs enter the queue at the back of the queue, and slowly make their way to the front as those ahead of them are run; but a job that enters the queue with a higher priority can `cut in line'.

The main factor which determines priority is whether or not the user (or their PI) has an [http://wiki.scinethpc.ca/wiki/index.php/Application_Process RAC allocation]. These are competitively allocated grants of computer time; there is a call for proposals towards the end of every calendar year. Users with an allocation have high priorities in an attempt to make sure that they can use the amount of computer time the committees granted them. Their priority decreases as they approach their allotted usage over the current window of time; by the time that they have exhausted that allotted usage, their priority is the same as users with no allocation (unallocated, or `default' users). Unallocated users have a fixed, low, priority.

This priority system is called `fairshare'; the scheduler attempts to make sure everyone has their fair share of the machines, where the share that's fair has been determined by the allocation committee. The fairshare window is a rolling window of one week; that is, any time you have a job in the queue, the fairshare calculation of its priority is given by how much of your allocation of the machine has been used in the last 7 days.

A particular allocation might have some fraction of GPC - say 4% of the machine (if the PI had been allocated 10 million CPU hours on GPC). The allocations have labels; (called `Resource Allocation Proposal Identifiers', or RAPIs) they look something like

abc-123-ab

where abc-123 is the PIs CCRI, and the suffix specifies which of the allocations granted to the PI is to be used. These can be specified on a job-by-job basis. On GPC, one adds the line
#PBS -A RAPI
to your script; on TCS, one uses
# @ account_no = RAPI
If the allocation to charge isn't specified, a default is used; each user has such a default, which can be changed at the same portal where one changes one's password,

https://portal.scinet.utoronto.ca/

A jobs priority is determined primarily by the fairshare priority of the allocation it is being charged to; the previous 7 days worth of use under that allocation is calculated and compared to the allocated fraction (here, 5%) of the machine over that window (here, 7 days). The fairshare priority is a decreasing function of the allocation left; if there is no allocation left (eg, jobs running under that allocation have already used 379,038 CPU hours in the past 7 days), the priority is the same as that of a user with no granted allocation. (This last part has been the topic of some debate; as the machine gets more utilized, it will probably be the case that we allow RAC users who have greatly overused their quota to have their priorities to drop below that of unallocated users, to give the unallocated users some chance to run on our increasingly crowded system; this would have no undue effect on our allocated users as they still would be able to use the amount of resources they had been allocated by the committees.) Note that all jobs charging the same allocation get the same fairshare priority.

There are other factors that go into calculating priority, but fairshare is the most significant. Other factors include
* length of time waiting in queue (measured in units of the requested runtime). A waiting queue job gains priority as it sits in the queue to avoid job starvation.
* User adjustment of priorities ( See below ).

The major effect of these subdominant terms is to shuffle the order of jobs running under the same allocation.

===How do we manage job priorities within our research group?===

'''Answer:'''

Obviously, managing shared resources within a large group - whether it
is conference funding or CPU time - takes some doing.

It's important to note that the fairshare periods are intentionally kept
quite short - just two weeks long. So, for example, let us say that in your resource
allocation you have about 10% of the machine. Then for someone to use
up the whole two week amount of time in 2 days, they'd have to use 70%
of the machine in those two days - which is unlikely to happen by
accident. If that does happen,
those using the same allocation as the person who used 70% of the
machine over the two days will suffer by having much lower priority for
their jobs, but only for the next 12 days - and even then, if there are
idle cpus they'll still be able to compute.

There will be online tools for seeing how the allocation is being used,
and those people who are in charge in your group will be able to use
that information to manage the users, telling them to dial it down or
up. We know that managing a large research group is hard, and we want
to make sure we provide you the information you need to do your job
effectively.

One way for users within a group to manage their priorities within the group
is with [[Moab#Adjusting_Job_Priority | user-adjusted priorities]]; this is
described in more detail on the [[Moab | Scheduling System]] page.



=== I couldn't find the .o output file in the .pbs_spool directory as I used to ===

On Feb 24 2011, the temporary location of standard input and output files was moved from the shared file system ${SCRATCH}/.pbs_spool to the
node-local directory /var/spool/torque/spool (which resides in ram). The final location after a job has finished is unchanged,
but to check the output/error of running jobs, users will now have to ssh into the (first) node assigned to the job and look in
/var/spool/torque/spool.

This alleviates access contention to the temporary directory, especially for those users that are running a lot of jobs, and reduces the burden on the file system in general.

Note that it is good practice to redirect output to a file rather than to count on the scheduler to do this for you.

=== My GPC job died, telling me `Copy Stageout Files Failed' ===

'''Answer:'''

When a job runs on GPC, the script's standard output and error are redirected to
<tt>$PBS_JOBID.gpc-sched.OU</tt> and <tt>$PBS_JOBID.gpc-sched.ER</tt> in
/var/spool/torque/spool on the (first) node on which your job is running. At the end of the job, those .OU and .ER files are copied to where the batch script tells them to be copied, by default <tt>$PBS_JOBNAME.o$PBS_JOBID</tt> and<tt>$PBS_JOBNAME.e$PBS_JOBID</tt>. (You can set those filenames to be something clearer with the -e and -o options in your PBS script.)

When you get errors like this:
<pre>
An error has occurred processing your job, see below.
request to copy stageout files failed on node
</pre>
it means that the copying back process has failed in some way. There could be a few reasons for this. The first thing to '''make sure that your .bashrc does not produce any output''', as the output-stageout is performed by bash and further output can cause this to fail.
But it also could have just been a random filesystem error, or it could be that your job failed spectacularly enough to shortcircuit the normal job-termination process (e.g. ran out of memory very quickly) and those files just never got copied.

Write to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] if your input/output files got lost, as we will probably be able to retrieve them for you (please supply at least the jobid, and any other information that may be relevant).

Mind you that it is good practice to redirect output to a file rather than depending on the job scheduler to do this for you.



===IB Memory Errors, eg <tt> reg_mr Cannot allocate memory </tt>===

Infiniband requires more memory than ethernet; it can use RDMA (remote direct memory access) transport for which it sets aside registered memory to transfer data.

In our current network configuration, it requires a _lot_ more memory, particularly as you go to larger process counts; unfortunately, that means you can't get around the "I need more memory" problem the usual way, by running on more nodes. Machines with different memory or
network configurations may exhibit this problem at higher or lower MPI
task counts.

Right now, the best workaround is to reduce the number and size of OpenIB queues, using XRC: with the OpenMPI, add the following options to your mpirun command:

<pre>
-mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32 -mca btl_openib_max_send_size 12288
</pre>

With Intel MPI, you should be able to do

<pre>
module load intelmpi/4.0.3.008
mpirun -genv I_MPI_FABRICS=shm:ofa -genv I_MPI_OFA_USE_XRC=1 -genv I_MPI_OFA_DYNAMIC_QPS=1 -genv I_MPI_DEBUG=5 -np XX ./mycode
</pre>

to the same end.

For more information see [[GPC MPI Versions]].

===My compute job fails, saying <tt>libpng12.so.0: cannot open shared object file</tt> or <tt>libjpeg.so.62: cannot open shared object file</tt>===

'''Answer:'''

To maximize the amount of memory available for compute jobs, the compute nodes have a less complete system image than the development nodes. In particular, since interactive graphics libraries like matplotlib and gnuplot are usually used interactively, the libraries for their use are included in the devel nodes' image but not the compute nodes.

Many of these extra libraries are, however, available in the "extras" module. So adding a "module load extras" to your job submission script - or, for overkill, to your .bashrc - should enable these scripts to run on the compute nodes.

==Monitoring jobs in the queue==

===Why hasn't my job started?===

'''Answer:'''

Use the moab command

<pre>
checkjob -v jobid
</pre>

and the last couple of lines should explain why a job hasn't started.

Please see [[Moab| Job Scheduling System (Moab) ]] for more detailed information

===How do I figure out when my job will run?===

'''Answer:'''

Please see [[Moab#Available_Resources| Job Scheduling System (Moab) ]]



===Running checkjob on my job gives me messages about JobFail and rejected===

Running checkjob on my job gives me messages that suggest my job has failed, as below: what did I do wrong?

<pre>
AName: test
State: Idle
Creds: user:xxxxxx group:xxxxxxxx account:xxxxxxxx class:batch_ib qos:ibqos
WallTime: 00:00:00 of 8:00:00
BecameEligible: Wed Jul 23 10:39:27
SubmitTime: Wed Jul 23 10:38:22
(Time Queued Total: 00:01:47 Eligible: 00:01:05)

Total Requested Tasks: 8

Req[0] TaskCount: 8 Partition: ALL
Opsys: centos6computeA Arch: --- Features: ---

Notification Events: JobFail

IWD: /scratch/x/xxxxxxxx/xxxxxxx/xxxxxxx
Partition List: torque,DDR
Flags: RESTARTABLE
Attr: checkpoint
StartPriority: 76
rejected for Opsys - (null)
rejected for State - (null)
rejected for Reserved - (null)
NOTE: job req cannot run in partition torque (available procs do not meet requirements : 0 of 8 procs found)
idle procs: 793 feasible procs: 0

Node Rejection Summary: [Opsys: 117][State: 2895][Reserved: 19]

NOTE: job violates constraints for partition SANDY (partition SANDY not in job partition mask)

NOTE: job violates constraints for partition GRAVITY (partition GRAVITY not in job partition mask)

rejected for State - (null)
NOTE:
</pre>

'''Answer:'''

The output from check job is a little cryptic in places, and if you are wondering why your job hasn't started yet, you might think that "rejection" and "JobFail" suggest that there's something wrong. But the above message is actually normal; you can use the <tt>showstart</tt> command on your job to get a (preliminary, subject to change) estimate as to when the job will start, and you'll find that it is in fact scheduled to start up in the near future.

In the above message:

* `Notification Events: JobFail` just means that, if notifications are enabled, you'll get a message if the job fails;
* `job req cannot run in partition torque` just means that the job cannot run just yet (that's why it's queued);
* `job req cannot run in dynamic partition DDR now (insufficient procs available: 0 < 8)` says why: there aren't processors available; and
* `job violates constraints for partition SANDY/GRAVITY` just means that the job isn't eligable to run in those paritcular (small) sections of the cluster.

that is, the above output is the normal and expected (if somewhat cryptic) explanation as to why the job is waiting - nothing to worry about.

===How can I monitor my running jobs on TCS?===

How can I monitor the load of TCS jobs?

'''Answer:'''

You can get more information with the command
/xcat/tools/tcs-scripts/LL/jobState.sh
which I alias as:
alias llq1='/xcat/tools/tcs-scripts/LL/jobState.sh'
If you run "llq1 -n" you will see a listing of jobs together with a lot of information, including the load.

===How can I check the memory usage from my jobs?===

How can I check the memory usage from my jobs?

'''Answer:'''

In many occasions it can be really useful to take a look at how much memory your job is using while it is running.
There a couple of ways to do so:

1) using some of the [https://wiki.scinet.utoronto.ca/wiki/index.php/SciNet_Command_Line_Utilities command line utilities] we have developed, e.g: by using the '''jobperf''' or '''jobtop''' utilities, it will allow you to check the job performance and head's node utilization respectively.

2) ''ssh'' into the nodes where your job is being run and check for memory usage and system stats right there. For instance, trying the 'top' or 'free' commands, in those nodes.

Also, it always a good a idea and strongly encouraged to inspect the standard output-log and error-log generated for your job submissions.
These files are named respectively: ''JobName.{o|e}jobIdNumber''; where ''JobName'' is the name you gave to the job (via the '-N' PBS flag) and ''JobIdNumber'' is the id number of the job.
These files are saved in the working directory after the job is finished, but they can be also accessed on real-time using the '''jobError''' and '''jobOutput''' [https://wiki.scinet.utoronto.ca/wiki/index.php/SciNet_Command_Line_Utilities command line utilities] available loading the ''extras'' module.

Other related topics to memory usage: 
[https://wiki.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Ram_Disk Using Ram Disk]
 
[https://wiki.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Memory_Configuration Different Memory Configuration nodes]
 
[https://wiki.scinet.utoronto.ca/wiki/index.php/FAQ#Monitoring_jobs_in_the_queue Monitoring Jobs in the Queue]
 
[https://wiki.scinet.utoronto.ca/wiki/images/a/a0/TechTalkJobMonitoring.pdf Tech Talk on Monitoring Jobs]

===Can I run cron jobs on devel nodes to monitor my jobs?===

Can I run cron jobs on devel nodes to monitor my jobs?

'''Answer:'''

No, we do not permit cron jobs to be run by users. To monitor the status of your jobs using a cron job running on your own machine, use the command

<pre>
ssh myusername@login.scinet.utoronto.ca "qstat -u myusername"
</pre>

or some variation of this command. Of course, you will need to have SSH keys setup on the machine running the cron job, so that password entry won't be necessary.

=== How does one check the amount of used CPU-hours in a project, and how does one get statistics for each user in the project? ===

'''Answer:'''

This information is available on the scinet portal,https://portal.scinet.utoronto.ca, See also [[SciNet Usage Reports]].



=== I couldn't find the .o output file in the .pbs_spool directory as I used to ===

On Feb 24 2011, the temporary location of standard input and output files was moved from the shared file system ${SCRATCH}/.pbs_spool to the
node-local directory /var/spool/torque/spool (which resides in ram). The final location after a job has finished is unchanged,
but to check the output/error of running jobs, users will now have to ssh into the (first) node assigned to the job and look in
/var/spool/torque/spool.

This alleviates access contention to the temporary directory, especially for those users that are running a lot of jobs, and reduces the burden on the file system in general.

Note that it is good practice to redirect output to a file rather than to count on the scheduler to do this for you.

=== My GPC job died, telling me `Copy Stageout Files Failed' ===

'''Answer:'''

When a job runs on GPC, the script's standard output and error are redirected to
<tt>$PBS_JOBID.gpc-sched.OU</tt> and <tt>$PBS_JOBID.gpc-sched.ER</tt> in
/var/spool/torque/spool on the (first) node on which your job is running. At the end of the job, those .OU and .ER files are copied to where the batch script tells them to be copied, by default <tt>$PBS_JOBNAME.o$PBS_JOBID</tt> and<tt>$PBS_JOBNAME.e$PBS_JOBID</tt>. (You can set those filenames to be something clearer with the -e and -o options in your PBS script.)

When you get errors like this:
<pre>
An error has occurred processing your job, see below.
request to copy stageout files failed on node
</pre>
it means that the copying back process has failed in some way. There could be a few reasons for this. The first thing to '''make sure that your .bashrc does not produce any output''', as the output-stageout is performed by bash and further output can cause this to fail.
But it also could have just been a random filesystem error, or it could be that your job failed spectacularly enough to shortcircuit the normal job-termination process (e.g. ran out of memory very quickly) and those files just never got copied.

Write to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] if your input/output files got lost, as we will probably be able to retrieve them for you (please supply at least the jobid, and any other information that may be relevant).

Mind you that it is good practice to redirect output to a file rather than depending on the job scheduler to do this for you.



===IB Memory Errors, eg <tt> reg_mr Cannot allocate memory </tt>===

Infiniband requires more memory than ethernet; it can use RDMA (remote direct memory access) transport for which it sets aside registered memory to transfer data.

In our current network configuration, it requires a _lot_ more memory, particularly as you go to larger process counts; unfortunately, that means you can't get around the "I need more memory" problem the usual way, by running on more nodes. Machines with different memory or
network configurations may exhibit this problem at higher or lower MPI
task counts.

Right now, the best workaround is to reduce the number and size of OpenIB queues, using XRC: with the OpenMPI, add the following options to your mpirun command:

<pre>
-mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32 -mca btl_openib_max_send_size 12288
</pre>

With Intel MPI, you should be able to do

<pre>
module load intelmpi/4.0.3.008
mpirun -genv I_MPI_FABRICS=shm:ofa -genv I_MPI_OFA_USE_XRC=1 -genv I_MPI_OFA_DYNAMIC_QPS=1 -genv I_MPI_DEBUG=5 -np XX ./mycode
</pre>

to the same end.

For more information see [[GPC MPI Versions]].

===My compute job fails, saying <tt>libpng12.so.0: cannot open shared object file</tt> or <tt>libjpeg.so.62: cannot open shared object file</tt>===

'''Answer:'''

To maximize the amount of memory available for compute jobs, the compute nodes have a less complete system image than the development nodes. In particular, since interactive graphics libraries like matplotlib and gnuplot are usually used interactively, the libraries for their use are included in the devel nodes' image but not the compute nodes.

Many of these extra libraries are, however, available in the "extras" module. So adding a "module load extras" to your job submission script - or, for overkill, to your .bashrc - should enable these scripts to run on the compute nodes.

-->

==Data on SciNet disks==

===How do I find out my disk usage?===

'''Answer:'''

The standard unix/linux utilities for finding the amount of disk space used by a directory are very slow, and notoriously inefficient on the GPFS filesystems that we run on the SciNet systems. There are utilities that very quickly report your disk usage:

The <tt>'''diskUsage'''</tt> command, available with the 'extras' module on the login nodes, datamovers and the GPC devel nodes, provides information in a number of ways on the home, scratch, and project file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time.
This information is only updated hourly!

More information about these filesystems is available at the [[Data_Management | Data_Management]].

===How do I transfer data to/from SciNet?===

'''Answer:'''

All incoming connections to SciNet go through relatively low-speed connections to the <tt>login.scinet</tt> gateways, so using scp to copy files the same way you ssh in is not an effective way to move lots of data. Better tools are described in our page on [[Data_Management#Data_Transfer | Data Transfer]].

===My group works with data files of size 1-2 GB. Is this too large to transfer by scp to login.scinet.utoronto.ca ?===

'''Answer:'''

Generally, occasion transfers of data less than 10GB is perfectly acceptible to so through the login nodes. See [[Data_Management#Data_Transfer | Data Transfer]].

===How can I check if I have files in /scratch that are scheduled for automatic deletion?===

'''Answer:'''

Please see [[Storage_Quickstart#Scratch_Disk_Purging_Policy | Storage At SciNet]]

===How to allow my supervisor to manage files for me using ACL-based commands?===

'''Answer:'''

Please see [[Data_Management#File.2FOwnership_Management_.28ACL.29 | File/Ownership Management]]

===Can we buy extra storage space on SciNet?===

'''Answer:'''
Yes, please see [[Data_Management#Buying_storage_space_on_GPFS_or_HPSS | Buying storage space on GPFS or HPSS ]] for more details.

===Can I transfer files between BGQ and HPSS?===

'''Answer:'''
Yes, please see [https://support.scinet.utoronto.ca/wiki/index.php/BGQ#Bridge_to_HPSS Bridge to HPSS ] for more details.

==Keep 'em Coming!==

===Next question, please===

Send your question to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]; we'll answer it asap!

Main Page

2018-05-25T01:20:55Z

Dgruner:

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==



{|style="width:65%"
|style="width:10%"|{{Up|Niagara|Niagara_Quickstart}}
|style="width:10%"|{{Up|HPSS|HPSS}}
|style="width:10%"|{{Up|BGQ|https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ}}
|-
|style="width:10%"|{{Up|P7|P7}}
|style="width:10%"|{{Down|P8|P8}}
|style="width:10%"|{{Up|SGC|SOSCIP_GPU}}
|-
|style="width:10%"|{{Up|Scheduler|Niagara_Quickstart#Submitting_jobs}}
|style="width:10%"|{{Up|External Network|External Network}}
|style="width:10%"|{{Up|File system|Niagara_Quickstart#Storage_and_quotas}}
|}

<div style="font-size:10pt;">
Current Messages:


* May 24, 2018: The data centre is under annual maintenance. All systems are offline. Systems are expected to be back late afternoon today; check for updates on this page.
* May 18, 2018: Announcement: Annual scheduled maintenance downtime: Thursday May 24, starting 7:00 AM
* May 16, 2018: Cooling restored, systems online
* May 16, 2018: Cooling issue at datacentre again, all systems down

</div>
|}

{|style="border-spacing: 10px;width: 95%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [[Niagara_Quickstart|Niagara cluster for large parallel jobs]]
* [[HPSS | HPSS archival storage]]
* [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ SOSCIP BlueGene/Q cluster]
* [[SOSCIP_GPU | SOSCIP GPU cluster]]
* [[P7|Experimental Power 7 cluster]]
* [[P8|Experimental Power 8 GPU cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://courses.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/channel/UC42CaO-AAQhwqa8RGzE3daQ SciNet's YouTube channel]
* [[Modules specific to Niagara]]
* [[Burst Buffer]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
|}

Main Page

2018-04-11T18:28:34Z

Dgruner: /* For Non-GPC Users */

= Niagara =

==System architecture==

<ul>
<li>Total of 60,000 Intel x86-64 cores.</li>
<li>1,500 Lenovo SD530 nodes</li>
<li>2x Intel Skylake 6148 CPUs (40 cores @2.4GHz per node) (with hyperthreading to 80 threads & AVX512).</li>
<li>3.02 PFlops delivered / 4.6 PFlops theoretical (would've been #42 on the TOP500 in Nov'18).</li>
<li>188 GiB / 202 GB RAM per node (at least 4 GiB/core for user jobs).</li>
<li>Operating system: Linux (CentOS 7).
<li>Interconnect: EDR InfiniBand, Dragonfly+ topology with Adaptive Routing</li>
1:1 up to 432 nodes, effectively 2:1 beyond that.</li>
<li>No GPUs, no local disk.</li>
<li>Replaces the General Purpose Cluster (GPC) and Tightly Coupled System (TCS).</li></ul>

== Migration to Niagara ==

=== Migration for Existing Users of the GPC ===

* Accounts, $HOME & $PROJECT of active GPC users transferred to Niagara (except dot-files in ~).
* Data stored in $SCRATCH will not be transfered automatically.
* Users are to clean up $SCRATCH on the GPC as much as possible (remember it's temporary data!). Then they can transfer what they need using datamover nodes. Let us know if you need help.
* To enable this transfer, there will be a short period during which you can have access to Niagara as well as to the GPC storage resources. This period will end no later than May 9, 2018.

=== For Non-GPC Users ===

<ul>
<li>Those of you new to SciNet, but with 2018 RAC allocations on Niagara, will have your accounts created and ready for you to login.</li>
<li>New, non-RAC users: we are still working out the procedure to get access. If you can't wait, for now, you can follow the old route of requesting a SciNet Consortium Account on the [https://ccdb.computecanada.ca/me/facilities CCDB site].</li></ul>

== Using Niagara: Logging in ==

As with all SciNet and CC (Compute Canada) compute systems, access to Niagara is via ssh (secure shell) only.

To access SciNet systems, first open a terminal window (e.g. MobaXTerm on Windows).

Then ssh into the Niagara login nodes with your CC credentials:

<source lang="bash">
$ ssh -Y MYCCUSERNAME@niagara.scinet.utoronto.ca</source>

or

<source lang="bash">$ ssh -Y MYCCUSERNAME@niagara.computecanada.ca</source>

* The Niagara login nodes are where you develop, edit, compile, prepare and submit jobs.
* These login nodes are not part of the Niagara compute cluster, but have the same architecture, operating system, and software stack.
* The optional <code>-Y</code> is needed to open windows from the Niagara command-line onto your local X server.
* To run on Niagara's compute nodes, you must submit a batch job.

== Storage Systems and Locations ==

=== Home and scratch ===

You have a home and scratch directory on the system, whose locations will be given by

<code>$HOME=/home/g/groupname/myccusername</code>

<code>$SCRATCH=/scratch/g/groupname/myccusername</code>

<source lang="bash">nia-login07:~$ pwd
/home/s/scinet/rzon

nia-login07:~$ cd $SCRATCH

nia-login07:rzon$ pwd
/scratch/s/scinet/rzon</source>

=== Project location ===

Users from groups with a RAC allocation will also have a project directory.

<code>$PROJECT=/project/g/groupname/myccusername</code>

'''''IMPORTANT: Future-proof your scripts'''''

Use the environment variables (HOME, SCRATCH, PROJECT) instead of the actual paths! The paths may change in the future.

=== Storage Limits on Niagara ===

{| class="wikitable"
! location
! quota
!align="right"| block size
! expiration time
! backed up
! on login
! on compute
|-
| $HOME
| 100 GB
|align="right"| 1 MB
|
| yes
| yes
| read-only
|-
| $SCRATCH
| 25 TB
|align="right"| 16 MB
| 2 months
| no
| yes
| yes
|-
| $PROJECT
| by group allocation
|align="right"| 16 MB
|
| yes
| yes
| yes
|-
| $ARCHIVE
| by group allocation
|align="right"|
|
| dual-copy
| no
| no
|-
| $BBUFFER
| ?
|align="right"| 1 MB
| very short
| no
| ?
| ?
|}

<ul>
<li>Compute nodes do not have local storage.</li>
<li>Archive space is on [https://wiki.scinet.utoronto.ca/wiki/index.php/HPSS HPSS].</li>
<li>Backup means a recent snapshot, not an achive of all data that ever was.</li>
<li><code>$BBUFFER</code> stands for the Burst Buffer, a functionality that is still being set up. This will be a faster parallel storage tier for temporary data.</li></ul>

=== Moving data ===

'''''Move amounts less than 10GB through the login nodes.'''''

* Only Niagara login nodes visible from outside SciNet.
* Use scp or rsync to niagara.scinet.utoronto.ca or niagara.computecanada.ca (no difference).
* This will time out for amounts larger than about 10GB.

'''''Move amounts larger than 10GB through the datamover nodes.'''''

* From a Niagara login node, ssh to <code>nia-datamover1</code> or <code>nia-datamover2</code>.
* Transfers must originate from this datamover.
* The other side (e.g. your machine) must be reachable from the outside.
* If you do this often, consider using Globus, a web-based tool for data transfer.

'''''Moving data to HPSS/Archive/Nearline using the scheduler.'''''

* [https://wiki.scinet.utoronto.ca/wiki/index.php/HPSS HPSS] is a tape-based storage solution, and is SciNet's nearline a.k.a. archive facility.
* Storage space on HPSS is allocated through the annual [https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions Compute Canada RAC allocation].

== Software and Libraries ==

=== Modules ===

Once you are on one of the login nodes, what software is already installed?

* Other than essentials, all installed software is made available using module commands.
* These set environment variables (<code>PATH</code>, etc.)
* Allows multiple, conflicting versions of a given package to be available.
* module spider shows the available software.

<source lang="bash">nia-login07:~$ module spider
---------------------------------------------------
The following is a list of the modules currently av
---------------------------------------------------
CCEnv: CCEnv

NiaEnv: NiaEnv/2018a

anaconda2: anaconda2/5.1.0

anaconda3: anaconda3/5.1.0

autotools: autotools/2017
autoconf, automake, and libtool

boost: boost/1.66.0

cfitsio: cfitsio/3.430

cmake: cmake/3.10.2 cmake/3.10.3

...</source>

<ul>
<li><code>module load <module-name></code>
use particular software</li>
<li><code>module purge</code>
remove currently loaded modules</li>
<li><code>module spider</code>
(or <code>module spider <module-name></code>)
list available software packages</li>
<li><code>module avail</code>
list loadable software packages</li>
<li><code>module list</code>
list loaded modules</li></ul>

On Niagara, there are really two software stacks:

<ol style="list-style-type: decimal;">
<li>A Niagara software stack tuned and compiled for this machine. This stack is available by default, but if not, can be reloaded with
<source lang="bash">module load NiaEnv</source></li>
<li>The same software stack available on Compute Canada's General Purpose clusters [https://docs.computecanada.ca/wiki/Graham Graham] and [https://docs.computecanada.ca/wiki/Cedar Cedar], compiled (for now) for a previous generation of CPUs:
<source lang="bash">module load CCEnv</source>
If you want the same default modules loaded as on Cedar and Graham, then afterwards also <code>module load StdEnv</code>.</li></ol>

Note: the <code>*Env</code> modules are '''''sticky'''''; remove them by <code>--force</code>.

=== Tips for loading software ===

<ul>
<li>We advise '''''against''''' loading modules in your .bashrc.
This could lead to very confusing behaviour under certain circumstances.</li>
<li>Instead, load modules by hand when needed, or by sourcing a separate script.</li>
<li>Load run-specific modules inside your job submission script.</li>
<li>Short names give default versions; e.g. <code>intel</code> → <code>intel/2018.2</code>.
It is usually better to be explicit about the versions, for future reproducibility.</li>
<li>Handy abbreviations:</li></ul>

<pre class="sh"> ml → module list
ml NAME → module load NAME # if NAME is an existing module
ml X → module X</pre>

* Modules sometimes require other modules to be loaded first. 
Solve these dependencies by using <code>module spider</code>.

=== Module spider ===

Oddly named, the module subcommand spider is the search-and-advice facility for modules.

<source lang="bash">nia-login07:~$ module load openmpi
Lmod has detected the error: These module(s) exist but cannot be loaded as requested: "openmpi"
Try: "module spider openmpi" to see how to load the module(s).</source>
<source lang="bash">nia-login07:~$ module spider openmpi
------------------------------------------------------------------------------------------------------
openmpi:
------------------------------------------------------------------------------------------------------
Versions:
openmpi/2.1.3
openmpi/3.0.1
openmpi/3.1.0rc3

------------------------------------------------------------------------------------------------------
For detailed information about a specific "openmpi" module (including how to load the modules) use
the module s full name.
For example:

$ module spider openmpi/3.1.0rc3
------------------------------------------------------------------------------------------------------</source>
<source lang="bash">nia-login07:~$ module spider openmpi/3.1.0rc3
------------------------------------------------------------------------------------------------------
openmpi: openmpi/3.1.0rc3
------------------------------------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "openmpi/3.1.0rc3"
module is available to load.

NiaEnv/2018a gcc/7.3.0
NiaEnv/2018a intel/2018.2
</source>
<source lang="bash">nia-login07:~$ module load NiaEnv/2018a intel/2018.2 # note: NiaEnv is usually already loaded
nia-login07:~$ module load openmpi/3.1.0rc3</source>
<source lang="bash">nia-login07:~$ module list
Currently Loaded Modules:
1) NiaEnv/2018a (S) 2) intel/2018.2 3) openmpi/3.1.0.rc3

Where:
S: Module is Sticky, requires --force to unload or purge</source>

== Can I Run Commercial Software? ==

* Possibly, but you have to bring your own license for it.
* SciNet and Compute Canada have an extremely large and broad user base of thousands of users, so we cannot provide licenses for everyone's favorite software.
* Thus, the only commercial software installed on Niagara is software that can benefit everyone: Compilers, math libraries and debuggers.
* That means no Matlab, Gaussian, IDL,
* Open source alternatives like Octave, Python, R are available.
* We are happy to help you to install commercial software for which you have a license.
* In some cases, if you have a license, you can use software in the Compute Canada stack.

== Compiling on Niagara: Example ==

<source lang="bash">nia-login07:~$ module list
Currently Loaded Modules:
1) NiaEnv/2018a (S)
Where:
S: Module is Sticky, requires --force to unload or purge

nia-login07:~$ module load intel/2018.2 gsl/2.4

nia-login07:~$ ls
main.c module.c

nia-login07:~$ icc -c -O3 -xHost -o main.o main.c
nia-login07:~$ icc -c -O3 -xHost -o module.o module.c
nia-login07:~$ icc -o main module.o main.o -lgsl -mkl

nia-login07:~$ ./main</source>
== Testing ==

You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.

<ul>
<li>Small test jobs can be run on the login nodes.
Rule of thumb: couple of minutes, taking at most about 1-2GB of memory, couple of cores.</li>
<li>You can run the the ddt debugger on the login nodes after <code>module load ddt</code>.</li>
<li>Short tests that do not fit on a login node, or for which you need a dedicated node, request an 
interactive debug job with the salloc command
<source lang="bash">nia-login07:~$ salloc -pdebug --nodes N --time=1:00:00</source>
where N is the number of nodes. The duration of your interactive debug session can be at most one hour, can use at most 4 nodes, and each user can only have one such session at a time.</li></ul>

== Submitting jobs ==

<ul>
<li>Niagara uses SLURM as its job scheduler.</li>
<li>You submit jobs from a login node by passing a script to the sbatch command:
<source lang="bash">nia-login07:~$ sbatch jobscript.sh</source></li>
<li>This puts the job in the queue. It will run on the compute nodes in due course.</li>
<li>Jobs will run under their group's RRG allocation, or, if the group has none, under a RAS allocation (previously called `default' allocation).</li></ul>

Keep in mind:

<ul>
<li>Scheduling is by node, so in multiples of 40-cores.</li>
<li>Maximum walltime is 24 hours.</li>
<li>Jobs must write to your scratch or project directory (home is read-only on compute nodes).</li>
<li>Compute nodes have no internet access.
Download data you need beforehand on a login node.</li></ul>

== Scheduling by Node ==

<ul>
<li>All job resource requests on Niagara are scheduled as a multiple of '''nodes'''.</li>
<li>The nodes that your jobs run on are exclusively yours.
<ul>
<li>No other users are running anything on them.</li>
<li>You can ssh into them to see how things are going.</li></ul>
</li>
<li>Whatever your requests to the scheduler, it will always be translated into a multiple of nodes.</li>
<li>Memory requests to the scheduler are of no use.
Your job gets N x 202GB of RAM if N is the number of nodes.</li>
<li>You should '''use all 40 cores on each of the nodes''' that your job uses.
You will be contacted if you don't, and we will help you get more science done.</li></ul>

== Hyperthreading: Logical CPUs vs. cores ==

* Hyperthreading, a technology that leverages more of the physical hardware by pretending there are twice as many logical cores than real once, is enabled on Niagara.
* So the OS and scheduler see 80 logical cores.
* 80 logical cores vs. 40 real cores typically gives about a 5-10% speedup (YMMV).

'''Because Niagara is scheduled by node, hyperthreading is actually fairly easy to use:'''

<ul>
<li>Ask for a certain number of nodes N for your jobs.</li>
<li>You know that you get 40xN cores, so you will use (at least) a total of 40xN mpi processes or threads.
(mpirun, srun, and the OS will automaticallly spread these over the real cores)</li>
<li>But you should also test if running 80xN mpi processes or threads gives you any speedup.</li>
<li>Regardless, your usage will be counted as 40xNx(walltime in years).</li></ul>

=== Example submission script (OpenMP) ===

<source lang="bash">#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=40
#SBATCH --time=1:00:00
#SBATCH --job-name openmp_job
#SBATCH --output=openmp_output_%j.txt

cd $SLURM_SUBMIT_DIR

module load intel/2018.2

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

./openmp_example
# or "srun ./openmp_example".
</source>
Submit this script with the command:
<source lang="bash">nia-login07:~$ sbatch openmp_job.sh</source>
* First line indicates that this is a bash script.
* Lines starting with <code>#SBATCH</code> go to SLURM.
* sbatch reads these lines as a job request (which it gives the name <code>openmp_job</code>) .
* In this case, SLURM looks for one node with 40 cores to be run inside one task, for 1 hour.
* Once it found such a node, it runs the script:
** Change to the submission directory;
** Loads modules;
** Sets an environment variable;
** Runs the <code>openmp_example</code> application.

=== Example submission script (MPI) ===

<source lang="bash">#!/bin/bash
#SBATCH --nodes=8
#SBATCH --ntasks=320
#SBATCH --time=1:00:00
#SBATCH --job-name mpi_job
#SBATCH --output=mpi_output_%j.txt

cd $SLURM_SUBMIT_DIR

module load intel/2018.2
module load openmpi/3.1.0rc3

mpirun ./mpi_example
# or "srun ./mpi_example"
</source>
Submit this script with the command:
<source lang="bash">nia-login07:~$ sbatch mpi_job.sh</source>
<ul>
<li>First line indicates that this is a bash script.</li>
<li>Lines starting with <code>#SBATCH</code> go to SLURM.</li>
<li>sbatch reads these lines as a job request (which it gives the name <code>mpi_job</code>)</li>
<li>In this case, SLURM looks for 8 nodes with 40 cores on which to run 320 tasks, for 1 hour.</li>
<li>Once it found such a node, it runs the script:
<ul>
<li>Change to the submission directory;</li>
<li>Loads modules;</li>
<li>Runs the <code>mpi_example</code> application.</li></ul>

</li></ul>

== Monitoring queued jobs ==

Once the job is incorporated into the queue, there are some command you can use to monitor its progress.

<ul>
<li><code>squeue</code> to show the job queue (<code>squeue -u $USER</code> for just your jobs);</li>
<li><code>squeue -j JOBID</code> to get information on a specific job
(alternatively, <code>scontrol show job JOBID</code>, which is more verbose).</li>
<li><code>squeue -j JOBID -o "%.9i %.9P %.8j %.8u %.2t %.10M %.6D %S"</code> to get an estimate for when a job will run.</li>
<li><code>scancel -i JOBID</code> to cancel the job.</li>
<li><code>sinfo -pcompute</code> to look at available nodes.</li>
<li>More utilities like those that were available on the GPC are under development.</li></ul>

== Data Management and I/O Tips ==

* $HOME, $SCRATCH, and $PROJECT all use the parallel file system called GPFS.
* Your files can be seen on all Niagara login and compute nodes.
* GPFS is a high-performance file system which provides rapid reads and writes to large data sets in parallel from many nodes.
* But accessing data sets which consist of many, small files leads to poor performance.
* Avoid reading and writing lots of small amounts of data to disk. 

* Many small files on the system would waste space and would be slower to access, read and write.
* Write data out in binary. Faster and takes less space.
* Burst buffer (to come) is better for i/o heavy jobs and to speed up checkpoints.

== Further information ==

=== Useful sites ===

* SciNet: https://www.scinet.utoronto.ca
* Niagara: https://docs.computecanada.ca/wiki/niagara
* System Status: https://wiki.scinet.utoronto.ca/wiki/index.php/System_Alerts
* Training: https://support.scinet.utoronto.ca/education

=== Support ===

* support@scinet.utoronto.ca
* niagara@computecanada.ca

Main Page

2018-04-09T19:25:48Z

Dgruner: /* Testing */

= Niagara =

==System architecture==

<ul>
<li>Total of 60,000 Intel x86-64 cores.</li>
<li>1,500 Lenovo SD530 nodes</li>
<li>2x Intel Skylake 6148 CPUs (40 cores @2.4GHz per node).
(with hyperthreading to 80 threads & AVX512)</li>
<li>3.02 PFlops delivered / 4.6 PFlops theoretical.
(would've been #42 on the TOP500 in Nov'18)</li>
<li>188 GiB / 202 GB RAM per node.
(at least 4 GiB/core for user jobs)</li>
<li>Operating system: Linux (CentOS 7).
<li>Interconnect: EDR InfiniBand, Dragonfly+ topology with Adaptive Routing</li>
1:1 up to 432 nodes, effectively 2:1 beyond that.</li>
<li>No GPUs, no local disk.</li>
<li>Replaces the General Purpose Cluster (GPC) and Tightly Coupled System (TCS).</li></ul>

== Migration to Niagara ==

=== Migration for Existing Users of the GPC ===

* Accounts, $HOME & $PROJECT of active GPC users transferred to Niagara (except dot-files in ~).
* Data stored in $SCRATCH will not be transfered automatically.
* Users are to clean up $SCRATCH on the GPC as much as possible (remember it's temporary data!). Then they can transfer what they need using datamover nodes. Let us know if you need help.
* To enable this transfer, there will be a short period during which you can have access to Niagara as well as to the GPC storage resources. This period will end no later than May 9, 2018.

=== For Non-GPC Users ===

<ul>
<li>Those of you new to SciNet, but with 2018 RAC allocations on Niagara, will have your accounts created and ready for you to login.</li>
<li>New, non-RAC users: we are still working out the procedure to get access.
If you can't wait, for now, you can follow the old route of requesting a SciNet
Consortium Account on the [https://ccsb.computecanada.ca CCDB site].</li></ul>

== Using Niagara: Logging in ==

As with all SciNet and CC compute systems, access to Niagara is via ssh (secure shell) only.

To access SciNet systems, first open a terminal window (e.g. MobaXTerm on Windows).

Then ssh into the Niagara login nodes with your CC credentials:

<source lang="bash">
$ ssh -Y MYCCUSERNAME@niagara.scinet.utoronto.ca</source>

or

<source lang="bash">$ ssh -Y MYCCUSERNAME@niagara.computecanada.ca</source>

* The Niagara login nodes are where you develop, edit, compile, prepare and submit jobs.
* These login nodes are not part of the Niagara compute cluster, but have the same architecture, operating system, and software stack.
* The optional <code>-Y</code> is needed to open windows from the Niagara command-line onto your local X server.
* To run on Niagara's compute nodes, you must submit a batch job.

== Storage Systems and Locations ==

=== Home and scratch ===

You have a home and scratch directory on the system, whose locations will be given by

<code>$HOME=/home/g/groupname/myccusername</code>

<code>$SCRATCH=/scratch/g/groupname/myccusername</code>

<source lang="bash">nia-login07:~$ pwd
/home/s/scinet/rzon

nia-login07:~$ cd $SCRATCH

nia-login07:rzon$ pwd
/scratch/s/scinet/rzon</source>

=== Project location ===

Users from groups with a RAC allocation will also have a project directory.

<code>$PROJECT=/project/g/groupname/myccusername</code>

'''''IMPORTANT: Future-proof your scripts'''''

Use the environment variables instead of the actual paths!

=== Storage Limits on Niagara ===

{| class="wikitable"
! location
! quota
!align="right"| block size
! expiration time
! backed up
! on login
! on compute
|-
| $HOME
| 100 GB
|align="right"| 1 MB
|
| yes
| yes
| read-only
|-
| $SCRATCH
| 25 TB
|align="right"| 16 MB
| 2 months
| no
| yes
| yes
|-
| $PROJECT
| by group allocation
|align="right"| 16 MB
|
| yes
| yes
| yes
|-
| $ARCHIVE
| by group allocation
|align="right"|
|
| dual-copy
| no
| no
|-
| $BBUFFER
| ?
|align="right"| 1 MB
| very short
| no
| ?
| ?
|}

<ul>
<li>Compute nodes do not have local storage.</li>
<li>Archive space is on HPSS.</li>
<li>Backup means a recent snapshot, not an achive of all data that ever was.</li>
<li><code>$BBUFFER</code> stands for the Burst Buffer, a functionality that is still being setup,
but this will be a faster parallel storage tier for temporary data.</li></ul>

=== Moving data ===

'''''Move amounts less than 10GB through the login nodes.'''''

* Only Niagara login nodes visible from outside SciNet.
* Use scp or rsync to niagara.scinet.utoronto.ca or niagara.computecanada.ca (no difference).
* This will time out for amounts larger than about 10GB.

'''''Move amounts larger than 10GB through the datamover node.'''''

* From a Niagara login node, ssh to <code>nia-datamover1</code>.
* Transfers must originate from this datamover.
* The other side (e.g. your machine) must be reachable from the outside.
* If you do this often, consider using Globus, a web-based tool for data transfer.

'''''Moving data to HPSS/Archive/Nearline using the scheduler.'''''

* HPSS is a tape-based storage solution, and is SciNet's nearline a.k.a. archive facility.
* Storage space on HPSS is controled through the annual RAC allocation.

== Software and Libraries ==

=== Modules ===

Once you are on one of the login nodes, what software is already installed?

* Other than essentials, all software installed using module commands.
* sets environment variables (<code>PATH</code>, etc.)
* Allows multiple, conflicting versions of package to be available.
* module spider shows available software.

<source lang="bash">nia-login07:~$ module spider
---------------------------------------------------
The following is a list of the modules currently av
---------------------------------------------------
CCEnv: CCEnv

NiaEnv: NiaEnv/2018a

anaconda2: anaconda2/5.1.0

anaconda3: anaconda3/5.1.0

autotools: autotools/2017
autoconf, automake, and libtool

boost: boost/1.66.0

cfitsio: cfitsio/3.430

cmake: cmake/3.10.2 cmake/3.10.3

...</source>

<ul>
<li><code>module load <module-name></code>
use particular software</li>
<li><code>module purge</code>
remove currently loaded modules</li>
<li><code>module spider</code>
(or <code>module spider <module-name></code>)
list available software packages</li>
<li><code>module avail</code>
list loadable software packages</li>
<li><code>module list</code>
list loaded modules</li></ul>

On Niagara, there are really two software stacks:

<ol style="list-style-type: decimal;">
<li>A Niagara software stack tuned and compiled for this machine. This stack is available by default, but if not, can be reloaded with
<source lang="bash">module load NiaEnv</source></li>
<li>The same software stack available on Compute Canada's General Purpose clusters Graham and Cedar, compiled (for now) for a previous generation of the CPUs:
<source lang="bash">module load CCEnv</source>
If you want the same default modules loaded as on Cedar and Graham, then afterwards also <code>module load StdEnv</code>.</li></ol>

Note: the <code>*Env</code> modules are '''''sticky'''''; remove them by <code>--force</code>.

=== Tips for loading software ===

<ul>
<li>We advise '''''against''''' loading modules in your .bashrc.
This could lead to very confusing behaviour under certain circumstances.</li>
<li>Instead, load modules by hand when needed, or by sourcing a separate script.</li>
<li>Load run-specific modules inside your job submission script.</li>
<li>Short names give default versions; e.g. <code>intel</code> <math>\rightarrow</math> <code>intel/2018.2</code>.
It is usually better to be explicit about the versions, for future reproducibility.</li>
<li>Handy abbreviations:</li></ul>

<pre class="sh"> ml → module list
ml NAME → module load NAME
ml X → module X</pre>

* Modules sometimes require other modules to be loaded first. 
Solve these dependencies by using <code>module spider</code>.

=== Module spider ===

Oddly named, the module subcommand spider is the search-and-advice facility for modules.

<source lang="bash">nia-login07:~$ module load openmpi
Lmod has detected the error: These module(s) exist but cannot be loaded as requested: "openmpi"
Try: "module spider openmpi" to see how to load the module(s).</source>
<source lang="bash">nia-login07:~$ module spider openmpi
------------------------------------------------------------------------------------------------------
openmpi:
------------------------------------------------------------------------------------------------------
Versions:
openmpi/2.1.3
openmpi/3.0.1
openmpi/3.1.0rc3

------------------------------------------------------------------------------------------------------
For detailed information about a specific "openmpi" module (including how to load the modules) use
the module s full name.
For example:

$ module spider openmpi/3.1.0rc3
------------------------------------------------------------------------------------------------------</source>
<source lang="bash">nia-login07:~$ module spider openmpi/3.1.0rc3
------------------------------------------------------------------------------------------------------
openmpi: openmpi/3.1.0rc3
------------------------------------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "openmpi/3.1.0rc3"
module is available to load.

NiaEnv/2018a gcc/7.3.0
NiaEnv/2018a intel/2018.2
</source>
<source lang="bash">nia-login07:~$ module load NiaEnv/2018a intel/2018.2 # note: NiaEnv is usually already loaded
nia-login07:~$ module load openmpi/3.1.0rc3</source>
<source lang="bash">nia-login07:~$ module list
Currently Loaded Modules:
1) NiaEnv/2018a (S) 2) intel/2018.2 3) openmpi/3.1.0.rc3

Where:
S: Module is Sticky, requires --force to unload or purge</source>
== Can I Run Commercial Software? ==

* Possibly, but you have to bring your own license for it.
* SciNet and Compute Canada have an extremely large and broad user base of thousands of users, so we cannot provide licenses for everyone's favorite software.
* Thus, the only commercial software installed and accessible is software that can benefit everyone: Compilers, math libraries and debuggers.
* That means no Matlab, Gaussian, IDL,
* Open source alternatives like Octave, Python, R are available.
* We are happy to help you to install commercial software for which you have a license.
* In some cases, if you have a license, you can use software in the Compute Canada stack.

== Compiling on Niagara: Example ==

<source lang="bash">nia-login07:~$ module list
Currently Loaded Modules:
1) NiaEnv/2018a (S)
Where:
S: Module is Sticky, requires --force to unload or purge

nia-login07:~$ module load intel/2018.2 gsl/2.4

nia-login07:~$ ls
main.c module.c

nia-login07:~$ icc -c -O3 -xHost -o main.o main.c
nia-login07:~$ icc -c -O3 -xHost -o module.o module.c
nia-login07:~$ icc -o main module.o main.o -lgsl -mkl

nia-login07:~$ ./main</source>
== Testing ==

You really should test your code before you submit it to the cluster to know if your code is correct and what kind of resources you need.

<ul>
<li>Small test jobs can be run on the login nodes.
Rule of thumb: couple of minutes, taking at most about 1-2GB of memory, couple of cores.</li>
<li>You can run the the ddt debugger on the login nodes after <code>module load ddt</code>.</li>
<li>Short tests that do not fit on a login node, or for which you need a dedicated node, request an 
interactive debug job with the salloc command
<source lang="bash">nia-login07:~$ salloc -pdebug --nodes N --time=1:00:00</source>
where N is the number of nodes. The duration of your interactive debug session can be at most one hour, can use at most N nodes, and each user can only have one such session at a time.</li></ul>

== Submitting jobs ==

<ul>
<li>Niagara uses SLURM as its job scheduler.</li>
<li>You submit jobs from a login node by passing a script to the sbatch command:
<source lang="bash">nia-login07:~$ sbatch jobscript.sh</source></li>
<li>This puts the job in the queue. It will run on the compute nodes in due course.</li>
<li>Jobs will run under their group's RRG allocation, or, if the group has none, under a RAS allocation (previously called `default' allocation).</li></ul>

Keep in mind:

<ul>
<li>Scheduling is by node, so in multiples of 40-cores.</li>
<li>Maximum walltime is 24 hours.</li>
<li>Jobs must write to your scratch or project directory (home is read-only on compute nodes).</li>
<li>Compute nodes have no internet access.
Download data you need beforehand on a login node.</li></ul>

=== Example submission script (OpenMP) ===

<source lang="bash">#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=40
#SBATCH --time=1:00:00
#SBATCH --job-name openmp_job
#SBATCH --output=openmp_output_%j.txt

cd $SLURM_SUBMIT_DIR

module load intel/2018.2

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

srun ./openmp_example</source>
<source lang="bash">nia-login07:~$ sbatch openmp_job.sh</source>
* First line indicates that this is a bash script.
* Lines starting with <code>#SBATCH</code> go to SLURM.
* sbatch reads these lines as a job request (which it gives the name <code>openmp_job</code>) .
* In this case, SLURM looks for one node with 40 cores to be run inside one task, for 1 hour.
* Once it found such a node, it runs the script:
** Change to the submission directory;
** Loads modules;
** Sets an environment variable;
** Runs the <code>openmp_example</code> application.

=== Example submission script (MPI) ===

<source lang="bash">#!/bin/bash
#SBATCH --nodes=8
#SBATCH --ntasks=320
#SBATCH --time=1:00:00
#SBATCH --job-name mpi_job
#SBATCH --output=mpi_output_%j.txt

cd $SLURM_SUBMIT_DIR

module load intel/2018.2
module load openmpi/3.1.0rc3

srun ./mpi_example</source>
<source lang="bash">nia-login07:~$ sbatch mpi_job.sh</source>
<ul>
<li>First line indicates that this is a bash script.</li>
<li>Lines starting with <code>#SBATCH</code> go to SLURM.</li>
<li>sbatch reads these lines as a job request (which it gives the name <code>mpi_job</code>)</li>
<li>In this case, SLURM looks for 8 nodes with 40 cores on which to run 320 tasks, for 1 hour.</li>
<li>Once it found such a node, it runs the script:
<ul>
<li>Change to the submission directory;</li>
<li>Loads modules;</li>
<li>Runs the <code>mpi_example</code> application.</li></ul>

</li></ul>

== Monitoring queued jobs ==

Once the job is incorporated into the queue, there are some command you can use to monitor its progress.

<ul>
<li><code>squeue</code> to show the job queue (<code>squeue -u $USER</code> for just your jobs);</li>
<li><code>squeue -j JOBID</code> to get information on a specific job
(alternatively, <code>scontrol show job JOBID</code>, which is more verbose).</li>
<li><code>squeue -j JOBID -o "%.9i %.9P %.8j %.8u %.2t %.10M %.6D %S"</code> to get an estimate for when a job will run.</li>
<li><code>scancel -i JOBID</code> to cancel the job.</li>
<li><code>sinfo -pcompute</code> to look at available nodes.</li>
<li>More utilities like those that were available on the GPC are under development.</li></ul>

== Data Management and I/O Tips ==

* $HOME, $SCRATCH, and $PROJECT all use the parallel file system called GPFS.
* Your files can be seen on all Niagara login and compute nodes.
* GPFS is a high-performance file system which provides rapid reads and writes to large data sets in parallel from many nodes.
* But accessing data sets which consist of many, small files leads to poor performance.
* Avoid reading and writing lots of small amounts of data to disk. 

* Many small files on the system would waste space and would be slower to access, read and write.
* Write data out in binary. Faster and takes less space.
* Burst buffer (to come) is better for i/o heavy jobs and to speed up checkpoints.

== Further information ==

=== Useful sites ===

* SciNet: https://www.scinet.utoronto.ca
* Niagara: https://docs.computecanada.ca/wiki/niagara
* System Status: https://wiki.scinet.utoronto.ca/wiki/index.php/System_Alerts
* Training: https://support.scinet.utoronto.ca/education

=== Support ===

* support@scinet.utoronto.ca
* niagara@computecanada.ca