Teach

From SciNet Users Documentation
Jump to: navigation, search
Teach Cluster
Ibm idataplex dx360 m4.jpg
Installed (orig Feb 2013), Oct 2018
Operating System Linux (Centos 7.4)
Number of Nodes 42
Interconnect Infiniband (QDR)
Ram/Node 64 Gb
Cores/Node 16
Login/Devel Node teach01 (from teach.scinet)
Vendor Compilers icc/gcc
Queue Submission slurm

Teaching Cluster

SciNet has assembled some older compute hardware into a small cluster provided primarily for teaching purposes. It is configured similarly to the production Niagara system, however uses repurposed hardware. This system should not be used for production work as such the queuing policies are designed to provide fast job turnover and limit the amount of resources one person can use at a time. Questions about its use or problems should be sent to support@scinet.utoronto.ca.

Specifications

The cluster consists of 42 repurposed x86_64 nodes each with 16 cores (from two octal core Intel XeonSandybridge E5-2650 CPUs) running at 2.0GHz with 64GB of RAM per node. The nodes are interconnected with 2.6:1 blocking QDR Infiniband for MPI communications and disk I/O to the SciNet Niagara filesystems. In total this cluster contains 672 cores.

Login/Devel Node

Login via ssh with your scinet account to teach.scinet.utoronto.ca, which will bring directly to teach01 the gateway/devel node for this cluster. From teach01 you can compile, do short tests, and submit your jobs to the queue.

Software Modules

  
  anaconda3/5.2.0
  blast+/2.7.1
  boost/1.66.0, boost/1.67.0
  cmake/3.12.3
  ddt/18.1.2, ddt/18.2
  fftw/3.3.7
  gcc/7.3.0
  gmp/6.1.2
  gnu-parallel/20180322
  gnuplot/5.2.2
  gsl/2.4
  hdf5/1.8.20
  hisat2/2.1.0
  htseq/0.11.1
  intel/2018.4
  intelmpi/2018.4
  lmdb/0.9.22
  mkl/2018.4
  netcdf/4.6.1
  openmpi/3.1.1
  oprofile/1.3.0
  orthofinder/2.2.7
  r/3.4.3-anaconda5.1.0, r/3.5.0, r/3.5.1
  samtools/1.8
  stringtie/1.3.5
  valgrind/3.14.0

Interactive jobs

the login node teach01. This node is shared between students of a number of different courses. Use this node to develop and compile code, to run short tests, and to submit computations to the scheduler.

For a short interactive sessions on a dedicated compute node of the teach cluster, use the 'debugjob' command.

teach01:~$ debugjob N

where N is the number of nodes, If N=1, this gives an interactive session of three hours, when N=4 (the maximum), it gives you 45 minutes.

Submit a Job

Teach uses SLURM as its job scheduler. More-advanced details of how to interact with the scheduler can be found on the Slurm page.

You submit jobs from a login node by passing a script to the sbatch command:

teach01:~scratch$ sbatch jobscript.sh

This puts the job in the queue. It will run on the compute nodes in due course.

In most cases, you will want to submit from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).

It is worth mentioning some differences between niagara and teach clusters:

  • Each teach cluster node has two CPUs with 8 cores each, a total of 16 cores per node (there is no hyperthreading). Make sure to adjust accordingly the flags --ntasks-per-node or --ntasks together with --nodes for the examples found at Slurm page.
  • The current slurm configuration of the teach cluster allocates compute resources by core as opposed to by node. That means your tasks might land on nodes that have other jobs running, i.e. they might share the node. If you want to avoid that, make sure to add the following directive in your submitting script: #SBATCH --exclusive. This forces your job to use the compute nodes exclusively.
  • The maximum walltime is currently set to 4 hours.
  • There are 2 queues available: Compute queue and debug queue. Their usage limits are listed on the table below.

Limits

There are limits to the size and duration of your jobs, the number of jobs you can run and the number of jobs you can have queued. It also matters in which 'partition' the jobs runs. 'Partitions' are SLURM-speak for use cases. You specify the partition with the -p parameter to sbatch or salloc, but if you do not specify one, your job will run in the compute partition, which is the most common case.

Usage Partition Running jobs Submitted jobs (incl. running) Min. size of jobs Max. size of jobs Min. walltime Max. walltime
Compute jobs compute 6 12 1 core 8 nodes (128 cores) 15 minutes 4 hours
Testing or troubleshooting debug 1 1 1 node (16 cores) 4 nodes (64 cores) N/A 1 hour

Within these limits, jobs will still have to wait in the queue. The waiting time depends on many factors such as the allocation amount, how much allocation was used in the recent past, the number of nodes and the walltime, and how many other jobs are waiting in the queue.