Burst Buffer
Revision as of 17:07, 9 August 2018 by Ejspence (talk | contribs) (Protected "Burst Buffer" ([Edit=Allow only administrators] (indefinite) [Move=Allow only administrators] (indefinite)))
The Niagara burst buffer is a very fast, very high performance alternative to /scratch, made of solid-state drives (SSD). The ideal use-cases for the burst buffer are jobs which involve a lot of IOPS (Input/Output operations), too many for the /scratch file system.
The setup of the Niagara Burst Buffer is evolving as we come to better understand how best to use this resource. The preliminary setup is as follows:
- To get access to the burst buffer a user must first request space on it. If you desire access, send an email, detailing your motivation for desiring access to the burst buffer, to support@scinet.utoronto.ca.
- Users with burst buffer access will have a directory created on that resource. The location is accessible using the $BBUFFER environment variable. The directory is accessible like $SCRATCH, from all nodes.
- A quota of 10 TBs will be set for each burst buffer user.
- Users should to be sure to copy any final files back to $SCRATCH at the end of the job. Unlike ramdisk (/dev/shm), the files will remain on the burst buffer between jobs. However, users should still endeavour to clean up after each job.
- The burst buffer is intended for temporary files. A very-short purging policy (possibly 48 hours) will be implemented in the future.
Users should test a burst buffer workflow using a short test job before using the burst buffer in production.