Data Management: Difference between revisions
No edit summary |
No edit summary |
||
| Line 2: | Line 2: | ||
= Performance = | = Performance = | ||
With the exception of archive, the SciNet file systems use [https://www.vastdata.com/ VAST], an all-flash NVMe parallel file system. It provides high throughput for large datasets read or written in parallel from many nodes, and it is designed to handle metadata-heavy workloads as well. Even so, very large numbers of small files are inefficient to manage: they consume inodes, stress the backup and archive systems, and are slower to transfer than a few large files. This is something you should keep in mind when planning your input/output strategy for runs on SciNet. | With the exception of archive, the SciNet file systems use [https://www.vastdata.com/ VAST], an all-flash NVMe parallel file system shared by Trillium and Balam. It provides high throughput for large datasets read or written in parallel from many nodes, and it is designed to handle metadata-heavy workloads as well. Even so, very large numbers of small files are inefficient to manage: they consume inodes, stress the backup and archive systems, and are slower to transfer than a few large files. This is something you should keep in mind when planning your input/output strategy for runs on SciNet. | ||
For instance, if you run multi-process jobs, having each process write to its own file is not a scalable I/O solution. The directory is locked by the first process that accesses it, so the other processes must wait. The code becomes considerably less parallel, and the file system may time out while waiting, causing your program to fail mysteriously. Consider using MPI-IO (part of the MPI-2 standard), which allows files to be opened simultaneously by different processes, or using a dedicated process for I/O to which all other processes send their data, and which subsequently writes this data to a single file. | For instance, if you run multi-process jobs, having each process write to its own file is not a scalable I/O solution. The directory is locked by the first process that accesses it, so the other processes must wait. The code becomes considerably less parallel, and the file system may time out while waiting, causing your program to fail mysteriously. Consider using MPI-IO (part of the MPI-2 standard), which allows files to be opened simultaneously by different processes, or using a dedicated process for I/O to which all other processes send their data, and which subsequently writes this data to a single file. | ||
Revision as of 15:31, 9 November 2026
Understanding the various file systems, and how to use them properly, is critical to optimizing your workflow and being a good SciNet citizen. This page describes the various Trillium file systems and how to use them properly.
Performance
With the exception of archive, the SciNet file systems use VAST, an all-flash NVMe parallel file system shared by Trillium and Balam. It provides high throughput for large datasets read or written in parallel from many nodes, and it is designed to handle metadata-heavy workloads as well. Even so, very large numbers of small files are inefficient to manage: they consume inodes, stress the backup and archive systems, and are slower to transfer than a few large files. This is something you should keep in mind when planning your input/output strategy for runs on SciNet.
For instance, if you run multi-process jobs, having each process write to its own file is not a scalable I/O solution. The directory is locked by the first process that accesses it, so the other processes must wait. The code becomes considerably less parallel, and the file system may time out while waiting, causing your program to fail mysteriously. Consider using MPI-IO (part of the MPI-2 standard), which allows files to be opened simultaneously by different processes, or using a dedicated process for I/O to which all other processes send their data, and which subsequently writes this data to a single file.
Purpose of each file system
Trillium and Balam access several different file systems. Note that not all of these file systems are available to all users.
/home ($HOME)
/home is intended for individual user files, and for common software or small datasets shared within a group, provided the quota is respected. Otherwise you may consider /scratch or /project. /home is read-only on the compute nodes and has daily backups.
/scratch ($SCRATCH)
/scratch is intended for temporary or transient files, checkpoint dumps, the results of your computations and simulations, or any material that can be easily recreated or reacquired. You may use scratch as well for any intermediate step in your workflow, provided it does not create too much I/O (input/output) or too many small files on this storage pool. Once you have your final results, those that you want to keep for the long term, you may migrate them to /project or /archive. /scratch has no backups. There is currently no automatic purging policy on Trillium, but scratch is not a long-term storage location.
/project ($PROJECT)
/project is available to groups whose PIs have a storage allocation, and is intended for common group software, large static datasets, or any material very costly to be reacquired or re-generated by the group, associated with jobs currently running on Trillium or Balam. Material on /project is expected to remain relatively immutable over time. You should think of $PROJECT as if it were $STATIC. Temporary or transient files should be kept on scratch, not project. High data turnover stresses the TSM backup system and consumes tapes long after the material has been deleted, because of backup retention policies and the extra versions kept of the same file. Even renaming top-level directories makes the system assume that a completely new directory tree has been created and the old one deleted, so think carefully about your naming convention in advance and stick with it. Users abusing the project file system and using it as scratch will be flagged and contacted. Note that /project is only available to groups with RAC allocation.
/archive ($ARCHIVE)
/archive is available to groups whose PIs have a storage allocation, and on Trillium and Balam it is the 'nearline' storage pool. It is used to offload semi-active material temporarily from any of the above file systems. In practice, users offload and recall material as part of their regular workflow, or when they reach their quotas on scratch or project. That material can remain on HPSS for a few months to a few years. Note that /archive is only available to groups with RAC allocation.
/dev/shm (RAM)
On the Trillium and Balam nodes ramdisk is available. Ramdisk is much faster than real disk. Up to 566 GiB of the node's memory (about 75% of usable RAM) may be used as a temporary local file system. This memory counts against your job's memory limit, which matters on shared GPU nodes. This is particularly useful in the early stages of migrating desktop-computing codes to an HPC platform, especially those that use a lot of file I/O (Input/Output). Using a lot of I/O is a bottleneck in large scale computing, especially on parallel file systems, since the files are synchronized across the whole network.
$SLURM_TMPDIR (RAM)
For consistency with the general purpose Alliance clusters (Nibi, Fir, Rorqual and Narval), the environment variable $SLURM_TMPDIR will be set on Trillium compute jobs. Note that this variable will point to RAMdisk, not to local hard drives. The $SLURM_TMPDIR directory is empty when your job starts, and its contents are deleted when the job finishes.
Note that Trillium and Balam compute nodes have no local disks, so $SLURM_TMPDIR lives in memory (ramdisk), in contrast to the general purpose systems of the Alliance (formerly Compute Canada), i.e., Fir, Nibi, Rorqual and Narval, where this variable points to a directory on a node-local SSD disk.
Quotas
You should understand what each file system is for and how to use it properly. The table below summarizes them.
| location | quota | expiration time | backed up | on login nodes | on compute nodes | |
|---|---|---|---|---|---|---|
| $HOME | 100 GB / 250,000 files per user | yes | yes | read-only | ||
| $SCRATCH | 25 TB / 6,000,000 files per user provided group quota is not reached | none at present | no | yes | yes | |
| groups of up to 4 users | 50TB for the group | |||||
| groups of up to 11 users | 125TB for the group | |||||
| groups of up to 28 users | 250TB for the group | |||||
| groups of up to 60 users | 400TB for the group | |||||
| groups with over 60 users | 500TB for the group | |||||
| $PROJECT | by group allocation | yes | yes | yes | ||
| $ARCHIVE | by group allocation | dual-copy | no | no | ||
- Inode vs. Space quota (PROJECT and SCRATCH)
- dynamic quota per group (SCRATCH)
- Compute nodes do not have local storage.
- Archive space is on HPSS, and is not accessible on the login, compute, or datamover nodes.
- Backup means a recent snapshot, not a replica of all data or version that ever was.
How much disk space do I have left?
The diskusage_report command, available on the Trillium login nodes and datamovers, reports how much space and how many files you are using on the home, scratch, project, and archive file systems. It can also report your group's usage, show how your usage has changed over a period, and plot your usage over time. See the Alliance storage page for an example of its output.
Scratch purging
There is currently no automatic purging policy for /scratch on Trillium. Scratch is still not a long-term storage location and has no backups: move anything you need to keep to /project or /archive. A purging policy may be introduced in the future; this page will be updated if that happens.
Handling large collections of files
There are a number of situations in which dealing with a format that encapsulates and compacts very large numbers of small files may be more performant and convenient than handling those small files in expanded format directly in the file system. These alternative file formats also save a lot of stress on the backup system, since tape is not designed to deal with a lot of small files. It also helps you avoid reaching your inode quota.
Please visit this page for more details:
https://docs.alliancecan.ca/wiki/Handling_large_collections_of_files
Backup policy
Our backups are version-based, not date-based:
- In general, we keep the two most recent versions of a file, one per day, as long as the file exists on the file system. Once a file is deleted, we expire the oldest version from the backup and keep the most recent one for 60 days. After that grace period, the remaining version is expired as well.
- We may have one or two versions of a file in the backup for over 10 years, provided the original has never been deleted from the file system.
- Conversely, there may be no backup if you created the file in the morning and deleted it in the afternoon, since the backup system runs once a day around midnight and never had a chance to capture it.
- If you generate several versions of a file during the day, only the most recent version before the backup runs is captured for that day.
Moving data
Data for analysis and final results must be moved to and from Trillium and Balam. There are several ways to do this.
Using rsync/scp
Move amounts less than 10 GB through the login nodes.
- Login nodes and datamovers are visible from outside SciNet.
- Use scp or rsync to trillium.scinet.utoronto.ca or trillium.alliancecan.ca (no difference).
- Transfers larger than about 10 GB will time out.
Move amounts larger than 10 GB through the datamover nodes.
- From a login node, ssh to
tri-dm{1,2,3,4}. From there you can transfer to or from the cluster. - Alternatively, you may also login/scp/rsync directly to the datamovers from the outside:
tri-dm1.scinet.utoronto.ca tri-dm2.scinet.utoronto.ca tri-dm3.scinet.utoronto.ca tri-dm4.scinet.utoronto.ca
- If you do this often, consider using Globus, a web-based tool for data transfer.
You can make at most 4 connections in a 2-minute window to the login or datamover nodes, so bundle your transfers: pass several files to scp or rsync, copy whole directories, or create a tar archive, transfer it, and extract it at the other end.
To transfer smaller files between Trillium and other Digital Research Alliance of Canada clusters, use the SSH agent forwarding flag -A when logging in to the other cluster. For example, to copy files to Trillium from Nibi use:
ssh -A USERNAME@nibi.alliancecan.ca
then perform the copy:
[USERNAME@l1.nibi ~]$ scp file USERNAME@trillium.alliancecan.ca:/scratch/USERNAME/
Using Globus
See the Alliance Globus documentation and Globus.
The Trillium file-system endpoint is "alliancecan#trillium"; the archive endpoint is "alliancecan#hpss".
Moving data to HPSS/Archive/Nearline
HPSS is for long-term storage of data.
- HPSS is a tape-based storage solution, and is SciNet's nearline a.k.a. archive facility.
- Storage space on HPSS is allocated through the annual Resource Allocation Competition (RAC) administered by the Digital Research Alliance of Canada.
File/Ownership management (ACL)
- By default at SciNet, users in the same group have read permission on each other's files, but not write permission.
- You can use an access control list (ACL) to let your supervisor (or another user within your group) manage files for you (i.e., create, move, rename, delete), while still retaining your access and permission as the original owner of the files/directories. You may also let users in other groups or whole other groups access (read, execute) your files using this same mechanism.