Teach

Teach Cluster
Installed	(orig Feb 2013), Oct 2018
Operating System	Linux (Centos 7.4)
Number of Nodes	42
Interconnect	Infiniband (QDR)
Ram/Node	64 Gb
Cores/Node	16
Login/Devel Node	teach01 (from teach.scinet)
Vendor Compilers	icc/gcc
Queue Submission	slurm

Teaching Cluster

SciNet has assembled some older compute hardware into a small cluster provided primarily for teaching purposes. It is configured similarly to the production Niagara system, however uses repurposed hardware. This system should not be used for production work as such the queuing policies are designed to provide fast job turnover and limit the amount of resources one person can use at a time. Questions about its use or problems should be sent to support@scinet.utoronto.ca.

Specifications

The cluster consists of 42 repurposed x86_64 nodes each with 16 cores (from two octal core Intel XeonSandybridge E5-2650 CPUs) running at 2.0GHz with 64GB of RAM per node. The nodes are interconnected with 2.6:1 blocking QDR Infiniband for MPI communications and disk I/O to the SciNet Niagara filesystems. In total this cluster contains 672 cores.

Login/Devel Node

Login via ssh with your scinet account to teach.scinet.utoronto.ca, which will bring directly to teach01 the gateway/devel node for this cluster. From teach01 you can compile, do short tests, and submit your jobs to the queue.

Software Modules

  
  anaconda3/5.2.0
  blast+/2.7.1
  boost/1.66.0, boost/1.67.0
  cmake/3.12.3
  ddt/18.1.2, ddt/18.2
  fftw/3.3.7
  gcc/7.3.0
  gmp/6.1.2
  gnu-parallel/20180322
  gnuplot/5.2.2
  gsl/2.4
  hdf5/1.8.20
  hisat2/2.1.0
  htseq/0.11.1
  intel/2018.4
  intelmpi/2018.4
  lmdb/0.9.22
  mkl/2018.4
  netcdf/4.6.1
  openmpi/3.1.1
  oprofile/1.3.0
  orthofinder/2.2.7
  r/3.4.3-anaconda5.1.0, r/3.5.0, r/3.5.1
  samtools/1.8
  stringtie/1.3.5
  valgrind/3.14.0

Interactive jobs

The login node teach01 is shared between students of a number of different courses. Use this node to develop and compile code, to run short tests, and to submit computations to the scheduler.

For a short interactive sessions on a dedicated compute node of the teach cluster, use the 'debugjob' command.

teach01:~$ debugjob N

where N is the number of nodes. An interactive session defaults to three hours when N=1, and 45 minutes when N=4 (the maximum number of nodes allowed for an interactive session by debugjob).

Submit a Job

Teach uses SLURM as its job scheduler. More-advanced details of how to interact with the scheduler can be found on the Slurm page.

You submit jobs from a login node by passing a script to the sbatch command:

teach01:~scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course.

In most cases, you will want to submit from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

It is worth mentioning some differences between niagara and teach clusters:

• Each teach cluster node has two CPUs with 8 cores each, a total of 16 cores per node (there is no hyperthreading). Make sure to adjust accordingly the flags --ntasks-per-node or --ntasks together with --nodes for the examples found at Slurm page.
• The current slurm configuration of the teach cluster allocates compute resources by core as opposed to by node. That means your tasks might land on nodes that have other jobs running, i.e. they might share the node. If you want to avoid that, make sure to add the following directive in your submitting script: #SBATCH --exclusive. This forces your job to use the compute nodes exclusively.
• The maximum walltime is currently set to 4 hours.
• There is only 1 queue available: the compute queue. Its usage limit is listed on the table below.
• 7 of the teach computing nodes have more memory than the 64GB default memory size. 5 of them have 128GB and 2 of them, 256GB. To run a big memory job on these nodes you need to add the following directive to your submitting script: #SBATCH --constraint=m128G. Replace m128G for m256G if you want your job to run exclusively on the 256GB nodes.

Limits

There are limits to the size and duration of your jobs, the number of jobs you can run and the number of jobs you can have queued. It also matters in which 'partition' the jobs runs. 'Partitions' are SLURM-speak for use cases. You specify the partition with the -p parameter to sbatch or salloc, but if you do not specify one, your job will run in the compute partition, which is the only case on teach cluster.

Usage	Partition	Running jobs	Submitted jobs (incl. running)	Min. size of jobs	Max. size of jobs	Min. walltime	Max. walltime
Compute jobs	compute	6	12	1 core	8 nodes (128 cores)	15 minutes	4 hours

Within these limits, jobs will still have to wait in the queue. The waiting time depends on many factors such as the allocation amount, how much allocation was used in the recent past, the number of nodes and the walltime, and how many other jobs are waiting in the queue.