Teach
| Teach Cluster | |
|---|---|
| Installed | (orig Feb 2013), Oct 2018 |
| Operating System | Linux (Centos 7.4) |
| Number of Nodes | 42 |
| Interconnect | Infiniband (QDR) |
| Ram/Node | 64 Gb |
| Cores/Node | 16 |
| Login/Devel Node | teach01 (from teach.scinet) |
| Vendor Compilers | icc/gcc |
| Queue Submission | slurm |
Teaching Cluster
SciNet has assembled some older compute hardware into a small cluster provided primarily for teaching purposes. It is configured similarly to the production Niagara system, however uses repurposed hardware. This system should not be used for production work as such the queuing policies are designed to provide fast job turnover and limit the amount of resources one person can use at a time. Questions about its use or problems should be sent to support@scinet.utoronto.ca.
Specifications
The cluster consists of 42 repurposed x86_64 nodes each with 16 cores (from two octal core Intel XeonSandybridge E5-2650 CPUs) running at 2.0GHz with 64GB of RAM per node. The nodes are interconnected with 2.6:1 blocking QDR Infiniband for MPI communications and disk I/O to the SciNet Niagara filesystems. In total this cluster contains 672 cores.
Login/Devel Node
Login via ssh with your scinet account to teach.scinet.utoronto.ca, which will bring directly to teach01 the gateway/devel node for this cluster. From teach01 you can compile, do short tests, and submit your jobs to the queue.
Software Modules
| Module | Versions (2018a) | Description |
|---|---|---|
| anaconda2 | 5.1.0 | Built to complement the rich, open source Python community, the Anaconda platform provides an enterprise-ready data analytics platform that empowers companies to adopt a modern open data science analytics architecture
|
| anaconda3 | 5.2.0 | R is a free software environment for statistical computing and graphics
|
| astral | 4.7.12 | ASTRAL is a tool for estimating an unrooted species tree given a set of unrooted gene trees
|
| bcftools | 1.8 | SAMtools is a suite of programs for interacting with high-throughput sequencing data
|
| bedtools | 2.27.1 | The BEDTools utilities allow one to address common genomics tasks such as finding feature overlaps and computing coverage
|
| blast+ | 2.7.1 | Basic Local Alignment Search Tool, or BLAST, is an algorithm for comparing primary biological sequence information, such as the amino-acid sequences of different proteins or the nucleotides of DNA sequences
|
| boost | 1.67.0 1.66.0 | Boost provides free peer-reviewed portable C++ source libraries
|
| bowtie2 | 2.3.4.3 | Bowtie 2 is an ultrafast and memory-efficient tool for aligning sequencing reads to long reference sequences
|
| bwa | 0.7.17 | Burrows-Wheeler Aligner (BWA) is an efficient program that aligns relatively short nucleotide sequences against a long reference sequence such as the human genome
|
| bwameth | 0.4.0 | Fast and accurate alignment of BS-Seq reads
|
| cmake | 3.12.3 | CMake, the cross-platform, open-source build system
|
| cutadapt | 2.1 | Cutadapt finds and removes adapter sequences, primers, poly-A tails and other types of unwanted sequence from your high-throughput sequencing reads
|
| deeptools | 3.2.1-anaconda2 | deepTools is a suite of python tools particularly developed for the efficient analysis of high-throughput sequencing data, such as ChIP-seq, RNA-seq or MNase-seq
|
| dexseq | 1.24.4 | Inference of differential exon usage in RNA sequencing
|
| fastqc | 0.11.8 | FastQC is a quality control application for high throughput sequence data
|
| fftw | 3.3.7 | FFTW is a C subroutine library for computing the discrete Fourier transform (DFT) in one or more dimensions, of arbitrary input size, and of both real and complex data
|
| gcc | 7.3.0 | The GNU Compiler Collection for C, C++, and Fortran
|
| git-annex | 2.8.1 | git-annex allows managing files with git, without checking the file contents into git
|
| gmp | 6.1.2 | GMP is a free library for arbitrary precision arithmetic, operating on signed integers, rational numbers, and floating point numbers
|
| gnu-parallel | 20180322 | GNU parallel is a shell tool for executing (usually serial) jobs in parallel
|
| gnuplot | 5.2.2 | Portable interactive, function plotting utility
|
| gsl | 2.4 | The GNU Scientific Library (GSL) is a numerical library for C and C++
|
| hdf5 | 1.8.20 1.10.4 | HDF5 is a data model, library, and file format for storing and managing data
|
| hisat2 | 2.1.0 | HISAT2 is a fast and sensitive alignment program for mapping next-generation sequencing reads (both DNA and RNA) against the general human population (as well as against a single reference genome)
|
| htseq | 0.11.1-anaconda2 0.11.1 | A framework to process and analyze data from high-throughput sequencing (HTS) assays
|
| htslib | 1.8 | A C library for reading/writing high-throughput sequencing data
|
| intel | 2018.4 | Intel compilers suite for C, C++, and Fortran, including the MKL, TBB, IPP, DAAL, and PSTL libraries
|
| intelmpi | 2018.4 | Intel MPI library with compiler wrappers for C, C++, and Fortran
|
| java | 1.8.0_201 | Java Platform, Standard Edition (Java SE) lets you develop and deploy Java applications on desktops and servers
|
| lmdb | 0.9.22 | OpenLDAP's Lightning Memory-Mapped Database (LMDB) library
|
| methyldackel | 0.4.0 | A (mostly) universal methylation extractor for BS-seq experiments
|
| metilene | 0.2.7 | Fast and sensitive detection of differential DNA methylation
|
| miso | 0.5.4 | A probabilistic framework that quantitates the expression level of alternatively spliced genes from RNA-Seq data, and identifies differentially regulated isoforms or exons across samples
|
| mkl | 2018.4 | Intel Math Kernel Library
|
| netcdf | 4.6.1 | NetCDF (network Common Data Form) is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data
|
| openmpi | 3.1.1 | The Open MPI Project is an open source MPI-2 implementation
|
| oprofile | 1.3.0 | OProfile is a system-wide profiler for Linux systems, capable of profiling all running code at low overhead
|
| orthofinder | 2.2.7 | Program for identifying orthologous protein sequence families
|
| pgplot | 5.2.2-x | Graphics subroutine library for C/C++ and Fortran
|
| prinseq | 0.20.4 | A bioinformatics tool to PRe-process and show INformation of SEQuence data
|
| r | 3.5.1 3.5.0 | R is a free software environment for statistical computing and graphics
|
| raxml | 8.2.12 | RAxML search algorithm for maximum likelihood based inference of phylogenetic trees
|
| samtools | 1.8 | SAM Tools provide various utilities for manipulating alignments in the SAM format, including sorting, merging, indexing and generating alignments in a per-position format
|
| singularity | 2.6.1 | Singularity is a portable application stack packaging and runtime utility.
|
| stringtie | 1.3.5 | StringTie is a fast and highly efficient assembler of RNA-Seq alignments into potential transcripts
|
| tbb | 2019.4 | Intel(R) Threading Building Blocks (Intel(R) TBB) lets you easily write parallel C++ programs that take full advantage of multicore performance, that are portable, composable and have future-proof scalability
|
| trimgalore | 0.6.0 | Function A wrapper tool around Cutadapt and FastQC to consistently apply quality and adapter trimming to FastQ files, with some extra functionality for MspI-digested RRBS-type (Reduced Representation Bisufite-Seq) libraries
|
| upsetr | 1.3.3 | R implementation of the UpSet set visualization technique published by Lex, Gehlenborg, et al
|
| valgrind | 3.14.0 | Valgrind provides debugging and profiling tools
|
Interactive jobs
The login node teach01 is shared between students of a number of different courses. Use this node to develop and compile code, to run short tests, and to submit computations to the scheduler.
For a interactive sessions on a compute node of the teach cluster, use the 'debugjob' command.
teach01:~$ debugjob -n C
where C is the number of cores. An interactive session defaults to three hours when using at most one node (C<=16), and becomes 45 minutes when using four nodes (i.e., 48<C<=64), which the maximum number of nodes allowed for an interactive session by debugjob.
For a short interactive sessions on a dedicated compute node of the teach cluster, use the 'debugjob' command.
teach01:~$ debugjob N
where N is the number of nodes. On the Teach cluster, this is equivalent to debugjob -n 16*N. N must be a positive integer and can at most be 4. .
Submit a Job
Teach uses SLURM as its job scheduler. More-advanced details of how to interact with the scheduler can be found on the Slurm page.
You submit jobs from a login node by passing a script to the sbatch command:
teach01:~scratch$ sbatch jobscript.sh
This puts the job in the queue. It will run on the compute nodes in due course.
In most cases, you will want to submit from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).
It is worth mentioning some differences between niagara and teach clusters:
- Each teach cluster node has two CPUs with 8 cores each, a total of 16 cores per node (there is no hyperthreading). Make sure to adjust accordingly the flags --ntasks-per-node or --ntasks together with --nodes for the examples found at Slurm page.
- The current slurm configuration of the teach cluster allocates compute resources by core as opposed to by node. That means your tasks might land on nodes that have other jobs running, i.e. they might share the node. If you want to avoid that, make sure to add the following directive in your submitting script: #SBATCH --exclusive. This forces your job to use the compute nodes exclusively.
- The maximum walltime is currently set to 4 hours.
- There is only 1 queue available: the compute queue. Its usage limit is listed on the table below.
- 7 of the teach computing nodes have more memory than the 64GB default memory size. 5 of them have 128GB and 2 of them, 256GB. To run a big memory job on these nodes you need to add the following directive to your submitting script: #SBATCH --constraint=m128G. Replace m128G for m256G if you want your job to run exclusively on the 256GB nodes.
Limits
There are limits to the size and duration of your jobs, the number of jobs you can run and the number of jobs you can have queued. It also matters in which 'partition' the jobs runs. 'Partitions' are SLURM-speak for use cases. You specify the partition with the -p parameter to sbatch or salloc, but if you do not specify one, your job will run in the compute partition, which is the only case on teach cluster.
| Usage | Partition | Running jobs | Submitted jobs (incl. running) | Min. size of jobs | Max. size of jobs | Min. walltime | Max. walltime |
|---|---|---|---|---|---|---|---|
| Compute jobs | compute | 6 | 12 | 1 core | 8 nodes (128 cores) | 15 minutes | 4 hours |
Within these limits, jobs will still have to wait in the queue. The waiting time depends on many factors such as the allocation amount, how much allocation was used in the recent past, the number of nodes and the walltime, and how many other jobs are waiting in the queue.