Teach

Teach Cluster
Installed	(orig Feb 2013), Oct 2018
Operating System	Linux (Centos 7.4)
Number of Nodes	42
Interconnect	Infiniband (QDR)
Ram/Node	64 Gb
Cores/Node	16
Login/Devel Node	teach01 (from teach.scinet)
Vendor Compilers	icc/gcc
Queue Submission	slurm

Teaching Cluster

SciNet has assembled some older compute hardware into a small cluster provided primarily for teaching purposes. It is configured similarly to the production Niagara system, however uses repurposed hardware. This system should not be used for production work as such the queuing policies are designed to provide fast job turnover and limit the amount of resources one person can use at a time. Questions about its use or problems should be sent to support@scinet.utoronto.ca.

Specifications

The cluster consists of 42 repurposed x86_64 nodes each with two octal core Intel Xeon (Sandybridge) E5-2650 2.0GHz CPUs with 64GB of RAM per node. The nodes are interconnected with 2.6:1 blocking QDR Infiniband for MPI communications and disk I/O to the SciNet Niagara filesystems. In total this cluster contains 672 x86_64 cores.

Login/Devel Node

Login via ssh with your scinet account to teach.scinet.utoronto.ca, which will bring directly to teach01 the gateway/devel node for this cluster. From teach01 you can compile, do short tests, and submit your jobs to the queue.

Software Modules

  anaconda3/5.2.0
  blast+/2.7.1
  boost/1.66.0, boost/1.67.0
  cmake/3.12.3
  ddt/18.1.2, ddt/18.2
  gcc/7.3.0
  gmp/6.1.2
  gnu-parallel/20180322
  hisat2/2.1.0
  htseq/0.11.1
  intel/2018.4
  intelmpi/2018.4
  lmdb/0.9.22
  mkl/2018.4
  openmpi/3.1.1
  oprofile/1.3.0
  r/3.4.3-anaconda5.1.0, r/3.5.0, r/3.5.1
  samtools/1.8
  stringtie/1.3.5
  valgrind/3.14.0

Submit a Job

Teach uses SLURM as its job scheduler. More-advanced details of how to interact with the scheduler can be found on the Slurm page.

You submit jobs from a login node by passing a script to the sbatch command:

teach01:~scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course.

In most cases, you will want to submit from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

It is worth mentioning some differences between niagara and teach clusters:

• Each teach cluster node has two CPUs with 8 cores each, a total of 16 cores per node (there is no hyperthreading). Make sure to adjust accordingly the flags --ntasks-per-node or --ntasks together with --nodes for the examples found at Slurm page.
• The current slurm configuration of the teach cluster allocates compute resources by core as opposed to by node. That means your tasks might land on nodes that have other jobs running, i.e. they might share the node. If you want to avoid that, make sure to add the following directive in your submitting script: #SBATCH --exclusive. This forces your job to use the compute nodes exclusively.
• The maximum walltime is currently set to 4 hours.
• There are 2 queues available: Compute queue and debug queue. Their usage limits are listed on the table below.

Limits

There are limits to the size and duration of your jobs, the number of jobs you can run and the number of jobs you can have queued. It also matters in which 'partition' the jobs runs. 'Partitions' are SLURM-speak for use cases. You specify the partition with the -p parameter to sbatch or salloc, but if you do not specify one, your job will run in the compute partition, which is the most common case.

Usage	Partition	Running jobs	Submitted jobs (incl. running)	Min. size of jobs	Max. size of jobs	Min. walltime	Max. walltime
Compute jobs	compute	6	12	1 core	8 nodes (128 cores)	15 minutes	4 hours
Testing or troubleshooting	debug	1	1	1 node (16 cores)	4 nodes (64 cores)	N/A	1 hour

Within these limits, jobs will still have to wait in the queue. The waiting time depends on many factors such as the allocation amount, how much allocation was used in the recent past, the number of nodes and the walltime, and how many other jobs are waiting in the queue.