SciNet Users Documentation - User contributions [en]

Main Page

2026-06-30T23:33:30Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up | Globus |Globus}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

''' Tue June 30th, 2026 15:35: ''' Cooling is back. Powering systems up

'''Tue June 30th, 2026 1:45 PM: ''' Repairs are still underway but are expected to be done later this afternoon.

'''Mon June 29th, 2026 6:40 PM: ''' Systems are down due to cooling emergency.

'''Fri Jun 26, 2026, 5:00 PM:''' Most systems are being brought back up.

'''Fri Jun 29, 2026, 1:00 PM:''' HPSS via Globus bug is fixed.

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-29T16:57:55Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Fri Jun 26, 2026, 5:00 PM:''' Most systems are being brought back up.

'''Fri Jun 29, 2026, 1:00 PM:''' HPSS via Globus bug is fixed.

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-27T13:52:57Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Fri Jun 26, 2026, 5:00 PM:''' Most systems are being brought back up.

'''Fri Jun 26, 2026, 4:00 PM:''' HPSS via Globus is currently affected by a bug during data ingestion (uploads into HPSS only).

* '''Action required for all users uploading to HPSS:'''
** In the Globus web interface, open '''Transfer & Sync Options''' in the transfer panel.
** Enable '''Do NOT verify file integrity after transfer'''.
** See the [[:File:Globus-HPSS-workaround.png|temporary workaround]] for reference.

* '''Without this setting:'''
** Only the first file in a transfer may be written to HPSS (sometimes the first two).
** Directory trees may be created, but remain incomplete (only partial ingestion).

* Downloads (recalls) from HPSS are not affected and continue to work normally.

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T22:38:25Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Fri Jun 26, 2026, 5:00 PM:''' Most systems are being brought back up.

'''Fri Jun 26, 2026, 4:00 PM:''' We upgraded Globus and HPSS to the latest software stack:
* alliancecan#trillium endpoint is working fine, no issues
* alliancecan#hpss currently we detected a bug, that makes the endpoint half broken:
** It will ingest only the first file of a sequence (sometimes the first 2), but nothing after that.
** It will create a whole directory tree, but with only 1 file, the rest of the tree is empty.
** Recalling multiple files or trees out of hpss is it working fine (this is the good half)
*** We have identified a [[:File:Globus-HPSS-workaround.png|temporary workaround]] for users who need to transfer data before a permanent fix is available:
*** Go to the "Transfer & Timer Options" in the middle of the file transfer pane.
*** find there the box to check called "do NOT verify file integrity after transfer" (forth checkbox down)
*** then you will be able to ingest multiple files and whole trees alliancecan#hpss

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T22:31:47Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Fri Jun 26, 2026, 5:00 PM:''' Most systems are being brought back up.

'''Fri Jun 26, 2026, 4:00 PM:''' We upgraded Globus and HPSS to the latest software stack:
* alliancecan#trillium endpoint is working fine, no issues
* alliancecan#hpss currently we detected a bug, that makes the endpoint half broken:
** It will ingest only the first file of a sequence (sometimes the first 2), but nothing after that.
** It will create a whole directory tree, but with only 1 file, the rest of the tree is empty.
** Recalling multiple files or trees out of hpss is it working fine (this is the good half)
*** We have identified a [[:File:Globus-HPSS-workaround.png|temporary workaround]] for users who need to transfer data before a permanent fix is available:
*** Go to the "Transfer & Timer Options" in the middle of the file transfer pane.
*** find there the box to check called "do NOT verify file integrity after transfer" (forth checkbox down)
*** then you will be able to ingest multiple files and whole trees

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T20:27:01Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}
'''Thu Jun 26, 2026, 4:00 PM:''' We upgraded Globus and HPSS to the latest software stack:
* alliancecan#trillium endpoint is working fine, no issues
* alliancecan#hpss currently we detected a bug, that makes the endpoint half broken:
** It will ingest only the first file of a sequence (sometimes the first 2), but nothing after that.
** It will create a whole directory tree, but with only 1 file, the rest of the tree is empty.
** Recalling multiple files or trees out of hpss is it working fine (this is the good half)
*** We have identified a [[:File:Globus-HPSS-workaround.png|temporary workaround]] for users who need to transfer data before a permanent fix is available:
*** Go to the "Transfer & Timer Options" in the middle of the file transfer pane.
*** find there the box to check called "do NOT verify file integrity after transfer" (forth checkbox down)
*** then you will be able to ingest multiple files and whole trees

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T20:24:46Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}
'''Thu Jun 26, 2026, 4:00 PM:''' We upgraded Globus and HPSS to the latest software stack:
* alliancecan#trillium endpoint is working fine, no issues
* alliancecan#hpss currently we detected a bug, that makes the endpoint half broken:
** It will ingest only the first file of a sequence (sometimes the first 2), but nothing after that.
** It will create a whole directory tree, but with only 1 file, the rest of the tree is empty.
** Recalling multiple files or trees out of hpss is it working fine (this is the good half)
** We came up with a [https://docs.scinet.utoronto.ca/index.php/File.png temporary workaround], if you plan to transfer things before we have a final solution:
*** Go to the "Transfer & Timer Options" in the middle of the file transfer pane.
*** find there the box to check called "do NOT verify file integrity after transfer" (forth checkbox down)
*** then you will be able to ingest multiple files and whole trees

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T20:23:34Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}
'''Thu Jun 26, 2026, 4:00 PM:''' We upgraded Globus and HPSS to the latest software stack:
* alliancecan#trillium endpoint is working fine, no issues
* alliancecan#hpss currently we detected a bug, that makes the endpoint half broken:
** It will ingest only the first file of a sequence (sometimes the first 2), but nothing after that.
** It will create a whole directory tree, but with only 1 file, the rest of the tree is empty.
** Recalling multiple files or trees out of hpss is it working fine (this is the good half)
** We came up with a [[https://docs.scinet.utoronto.ca/index.php/File:Globus-HPSS-workaround.png|temporary workaround]], if you plan to transfer things before we have a final solution:
*** Go to the "Transfer & Timer Options" in the middle of the file transfer pane.
*** find there the box to check called "do NOT verify file integrity after transfer" (forth checkbox down)
*** then you will be able to ingest multiple files and whole trees

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T20:22:58Z

Pinto:

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}
'''Thu Jun 26, 2026, 4:00 PM:''' We upgraded Globus and HPSS to the latest software stack:
* alliancecan#trillium endpoint is working fine, no issues
* alliancecan#hpss currently we detected a bug, that makes the endpoint half broken:
** It will ingest only the first file of a sequence (sometimes the first 2), but nothing after that.
** It will create a whole directory tree, but with only 1 file, the rest of the tree is empty.
** Recalling multiple files or trees out of hpss is it working fine (this is the good half)
** We came up with a [[Globus-HPSS-workaround.png|temporary workaround]], if you plan to transfer things before we have a final solution:
*** Go to the "Transfer & Timer Options" in the middle of the file transfer pane.
*** find there the box to check called "do NOT verify file integrity after transfer" (forth checkbox down)
*** then you will be able to ingest multiple files and whole trees

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

File:Globus-HPSS-workaround.png

2026-06-26T20:20:32Z

Pinto:

Main Page

2026-06-26T20:18:39Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}
'''Thu Jun 26, 2026, 4:00 PM:''' We upgraded Globus and HPSS to the latest software stack:
* alliancecan#trillium endpoint is working fine, no issues
* alliancecan#hpss currently we detected a bug, that makes the endpoint half broken:
** It will ingest only the first file of a sequence (sometimes the first 2), but nothing after that.
** It will create a whole directory tree, but with only 1 file, the rest of the tree is empty.
** Recalling multiple files or trees out of hpss is it working fine (this is the good half)
** We came up with a temporary workaround, if you plan to transfer things before we have a final solution:
*** Go to the "Transfer & Timer Options" in the middle of the file transfer pane.
*** find there the box to check called "do NOT verify file integrity after transfer" (forth checkbox down)
*** then you will be able to ingest multiple files and whole trees

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T20:18:13Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}
'''Thu Jun 25, 2026, 4:00 PM:''' We upgraded Globus and HPSS to the latest software stack:
* alliancecan#trillium endpoint is working fine, no issues
* alliancecan#hpss currently we detected a bug, that makes the endpoint half broken:
** It will ingest only the first file of a sequence (sometimes the first 2), but nothing after that.
** It will create a whole directory tree, but with only 1 file, the rest of the tree is empty.
** Recalling multiple files or trees out of hpss is it working fine (this is the good half)
** We came up with a temporary workaround, if you plan to transfer things before we have a final solution:
*** Go to the "Transfer & Timer Options" in the middle of the file transfer pane.
*** find there the box to check called "do NOT verify file integrity after transfer" (forth checkbox down)
*** then you will be able to ingest multiple files and whole trees

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T20:17:24Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}
'''Thu Jun 25, 2026, 4:00 PM:''' We upgraded Globus and HPSS to the lasted software stack:
* alliancecan#trillium endpoint is working fine, no issues
* alliancecan#hpss currently we detected a bug, that makes the endpoint half broken:
** It will ingest only the first file of a sequence (sometimes the first 2), but nothing after that.
** It will create a whole directory tree, but with only 1 file, the rest of the tree is empty.
** Recalling multiple files or trees out of hpss is it working fine (this is the good half)
** We came up with a temporary workaround, if you plan to transfer things before we have a final solution:
*** Go to the "Transfer & Timer Options" in the middle of the file transfer pane.
*** find there the box to check called "do NOT verify file integrity after transfer" (forth checkbox down)
*** then you will be able to ingest multiple files and whole trees

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T20:14:40Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}
'''Thu Jun 25, 2026, 4:00 PM:''' We upgraded Globus and HPSS to the lasted software stack:
* alliancecan#trillium endpoint is working fine, no issues
* alliancecan#hpss currently we detected a bug, that makes the endpoint half broken:
** It will ingest only the first file of a sequence (sometimes the first 2), but nothing after that.
** It will create a whole directory tree, but with only 1 file, the rest of the tree is empty.
** Recalling multiple files or trees out of hpss is it working fine (this is the good half)
** We came up with a temporary workaround, if you plan to transfer things before we have a final solution:
*** Go to the "Transfer & Timer Options" in the middle of the file transfer pane.
*** find there the box to check called "do NOT verify file integrity after transfer" (forth checkbox down)

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T20:09:44Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}
'''Thu Jun 25, 2026, 4:00 PM:''' We upgraded Globus and HPSS to the lasted software stack:
* alliancecan#trillium endpoint is working fine, no issues
* alliancecan#hpss currently we detected a bug, that makes the endpoint half broken:
** It will ingest only the first file of a sequence (sometimes the first 2), but nothing after that.
** It will create a whole directory tree, but with only 1 file, the rest of the tree is empty.
** Recalling multiple files or trees out of hpss is it working fine (this is the good half)
** We came up with a temporary workaround, if you plan to transfer things before we have a final solution:
*** Go to the "Transfer & Timer Options" in the middle of the file transfer pane.
*** find there the box to check called "do NOT verify file integrity after transfer" (forth checkbox down)

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T20:00:50Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up| Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T13:18:34Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Partial | Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T00:18:51Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Partial3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Partial | Globus |Globus}}
|-
|{{Partial | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-26T00:18:18Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Partial3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Partial | Globus |Globus}}
|-
|{{Down | HPSS|HPSS}}
|{{Down | Balam|Balam}}
|{{Down | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Thu Jun 25, 2026, 3:39 PM:''' Due to an unexpected technical issue, the SciNet systems can not yet be brought up today (except for the login nodes). We are working to have the systems back up tomorrow.

'''Tue Jun 23, 2026, 1:34 PM:''' Trillium login nodes and datamover nodes are up again, as is the OnDemand interface; This is mainly to give users access to their files; no jobs can be submittted for at least another day.

'''Trillium AI Expansion Installation Maintenance Shutdown Schedule'''

# '''May 26/27, 2026:''' Shutdown of the Trillium compute nodes and HPSS, starting at 4 AM EDT on May 26th. The Trillium login nodes as well as OnDemand, the Teach cluster, and Balam, will remain available during this maintenance. HPSS will be back in service later on the same day (May 26th), while the Trillium compute nodes are expected to be back in service at some time on May 27th.
# '''June 9/10, 2026:''' A multi-day chiller maintenance; this will involve a shutdown of all systems.
# '''June 22-25, 2026: A 4-day full datacentre shutdown.''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Balam, S4H, HPSS, Teach, as well as hosted equipment).
# '''July 13-24, 2026 (tentative):''' A two-week long shutdown of Trillium compute nodes will be required. During most of this time the Trillium login nodes, storage and Open Ondemand will remain up, except for a one-day power shutdown within this period (currently planned for July 22nd).

More information and precise dates for the last maintenance shutdown will be announced later.

'''Wed Apr 30, 2026, 3:00 pm:''' System have been updated to mitigate known security risks, and are back in service. Note that no actual security breaches were found.

'''Wed Apr 29, 2026, 5:25 pm:''' For security reasons, login access to all systems has been disabled, as have OnDemand Apps. Compute jobs are still allowed to run.

'''Thu Apr 23, 2026, 10:00 am:''' The Trillium file system has issues and may be slow on certain nodes. We are still investigating.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-06-09T18:55:51Z

Pinto: /* System Status */

Main Page

2026-05-27T20:23:51Z

Pinto: /* System Status */

Main Page

2026-04-07T02:12:20Z

Pinto: /* System Status */

Main Page

2026-04-07T02:12:01Z

Pinto: /* System Status */

Main Page

2026-04-06T13:14:32Z

Pinto: /* System Status */

Main Page

2026-04-02T17:54:58Z

Pinto:

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up | Globus |Globus}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Mon Apr 06, 2026, 8:00 pm:''' HPSS scheduled maintenance: update of HPSS to v11.3_u4 and hsi-htar to v11.3_u1 (bug fixes)

'''Wed Mar 25, 2026, 5:00 pm:''' Trillium is operational again.

'''Wed Mar 25, 2026, 9:00 am:''' Teach is operational again.

'''Tue Mar 24, 2026, 8:45 pm:''' Open OnDemand is operational again.

'''Tue Mar 24, 2026, 1:00 pm:''' External connectivity is back.

'''Tue Mar 24, 2026, 12:05 pm:''' External connectivity to the data centre was lost.

'''Tue Mar 24, 2026, 7:00 am:''' Maintenance has started.

'''Mon Mar 16, 2026. 13:30pm''' Recovering. Almost all systems are up again. Please resubmit your jobs that crashed.

'''Mon Mar 16, 2026. 12:00pm''' Power glitch at the data centre caused compute nodes to go down.

'''Thu Mar 12, 2026, 4:15 pm''' Connection to Trillium are operational again.

'''Thu Mar 12, 2026, 1:00 pm''' We've had some login issues particularly for Trillium-GPU. We're investigating.

'''Downtime Announcement:''' The winter cooling tower maintenance for the SciNet data centre will take place on March 24 and 25, 2026, starting at 7:00 a.m. on the 24th. All SciNet systems (Trillium, OnDemand, Balam, S4H, Teach, as well as hosted equipment) will have their compute nodes shut down. Login nodes, file systems, and the HPSS system will remain available, and
jobs will be held in the queue until maintenance is complete. Starting 7am on Mar 23, users are encouraged to submit small and short jobs that may be scheduled before the maintenance begins.

'''Fri Feb 20, 2026, 11:35 pm:''' Power glitch, ~480 compute nodes rebooted. Regional power quality has been quite poor lately ([https://www.yorkregion.com/news/road-salt-blamed-for-power-outages/article_1a36d25d-5f97-56ee-a0c7-c49c7b732d38.html 1],
[https://www.yorkregion.com/news/power-company-executive-responds-to-york-region-outages/article_c4d072e7-2892-5c9c-8deb-ac5e1936779c.html 2]).

'''Thu Feb 19, 2026, 3:00 pm:''' Systems restored. Please report issues to support@scinet.utoronto.ca.

'''Tue Feb 17, 2026, 8:40 am:''' Power outage at the data centre. Cooling issues have developed as a result. Major systems (Trillium, S4H) are expected to be down until sometime Thursday. Login nodes and file systems will remain accessible.

'''Mon Feb 16, 2026, 8:40 pm:''' Electricity is unstable in the data centre area due to severe snowfall.

'''Thu Jan 29, 2026, 1:40 pm:''' All services are operational again.

'''Thu Jan 29, 2026, 12:00 pm:''' The Trillium and Open OnDemand compute nodes are operational again. We are still working on bringing Balam, Neptune and S4H nodes up again.

'''Thu Jan 29, 2026, 10:00 am:''' There was a power glitch at the data centre overnight. The login nodes are accessible but the compute nodes are down.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-03-26T13:51:50Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up | Globus |Globus}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Wed Mar 25, 2026, 5:00 pm:''' Trillium is operational again.

'''Wed Mar 25, 2026, 9:00 am:''' Teach is operational again.

'''Tue Mar 24, 2026, 8:45 pm:''' Open OnDemand is operational again.

'''Tue Mar 24, 2026, 1:00 pm:''' External connectivity is back.

'''Tue Mar 24, 2026, 12:05 pm:''' External connectivity to the data centre was lost.

'''Tue Mar 24, 2026, 7:00 am:''' Maintenance has started.

'''Mon Mar 16, 2026. 13:30pm''' Recovering. Almost all systems are up again. Please resubmit your jobs that crashed.

'''Mon Mar 16, 2026. 12:00pm''' Power glitch at the data centre caused compute nodes to go down.

'''Thu Mar 12, 2026, 4:15 pm''' Connection to Trillium are operational again.

'''Thu Mar 12, 2026, 1:00 pm''' We've had some login issues particularly for Trillium-GPU. We're investigating.

'''Downtime Announcement:''' The winter cooling tower maintenance for the SciNet data centre will take place on March 24 and 25, 2026, starting at 7:00 a.m. on the 24th. All SciNet systems (Trillium, OnDemand, Balam, S4H, Teach, as well as hosted equipment) will have their compute nodes shut down. Login nodes, file systems, and the HPSS system will remain available, and
jobs will be held in the queue until maintenance is complete. Starting 7am on Mar 23, users are encouraged to submit small and short jobs that may be scheduled before the maintenance begins.

'''Fri Feb 20, 2026, 11:35 pm:''' Power glitch, ~480 compute nodes rebooted. Regional power quality has been quite poor lately ([https://www.yorkregion.com/news/road-salt-blamed-for-power-outages/article_1a36d25d-5f97-56ee-a0c7-c49c7b732d38.html 1],
[https://www.yorkregion.com/news/power-company-executive-responds-to-york-region-outages/article_c4d072e7-2892-5c9c-8deb-ac5e1936779c.html 2]).

'''Thu Feb 19, 2026, 3:00 pm:''' Systems restored. Please report issues to support@scinet.utoronto.ca.

'''Tue Feb 17, 2026, 8:40 am:''' Power outage at the data centre. Cooling issues have developed as a result. Major systems (Trillium, S4H) are expected to be down until sometime Thursday. Login nodes and file systems will remain accessible.

'''Mon Feb 16, 2026, 8:40 pm:''' Electricity is unstable in the data centre area due to severe snowfall.

'''Thu Jan 29, 2026, 1:40 pm:''' All services are operational again.

'''Thu Jan 29, 2026, 12:00 pm:''' The Trillium and Open OnDemand compute nodes are operational again. We are still working on bringing Balam, Neptune and S4H nodes up again.

'''Thu Jan 29, 2026, 10:00 am:''' There was a power glitch at the data centre overnight. The login nodes are accessible but the compute nodes are down.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-01-17T04:12:02Z

Pinto:

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up | Globus |Globus}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Thu Jan 16, 2026, 11:00 pm:''' HPSS is back online, and accessible via alliancecan#hpss Globus endpoint.

'''Thu Jan 15, 2026, 10:00 pm:''' HPSS will undergo maintenance on Friday morning, Jan/16/2025, , including alliancecan#hpss Globus endpoint

'''Tue Jan 6, 2026, 10:15 am:''' OnDemand has been fixed and is working again.

'''Mon Jan 5, 2026, 9:00 pm:''' The authentication mechanism of OnDemand is not working.

'''Wed Dec 31, 2025, 12:40 pm:''' We believe the problem has now been resolved. Please let us know if you still experience login problems or aborted jobs.

'''Tue Dec 30, 2025, 2:10 pm:''' We are experiencing problems with authentication, resulting in failed logins, OOD errors, and aborted jobs (with "prolog error"). Please bear with us, as we are very short-staffed during the holiday break. We will post updates here.

'''Tue Dec 3, 2025, 11:30 am:''' Open OnDemand is fully operational again.

'''Sat Nov 29, 2025, 00:40 am:''' There has been a problem with the water chiller. Some systems are offline.

'''Wed Nov 5, 2025, 12:55 pm:''' Balam is back online.

'''Wed Nov 5, 2025, 10:00 am:''' Open OnDemand is back online.

'''Tue Nov 4, 2025, 11:00 pm:''' Most of the work is done, data movers, Globus, and HPSS are back online. Remaining services will be worked on tomorrow.

'''Tue Nov 4, 2025, 8:30 am:''' Scheduled network maintenance. Trillium cluster is *not* affected.

'''Tue Oct 21, 2025, 17:30 am:''' Balam maintenance finished.

'''Tue Oct 21, 2025, 7:00 am:''' Balam maintenance day.

'''Thu Oct 15, 2025, 3:55 pm:''' Trillium inbound connections through trillium.alliancecan.ca or trillium.scinet.utoronto.ca are working again.

'''Thu Oct 15, 2025, 3:05 pm:''' Trillium is experiencing external network issues for both incoming traffic. Please try: ssh USERNAME@tri-login01.scinet.utoronto.ca in the meantime.

'''Thu Oct 06, 2025, 8:00 pm:''' HPSS is fully functional. You may submit archive jobs from trillium login nodes, datamovers and robots.

'''Thu Oct 03, 2025, 6:30 pm:''' HPSS is back online, and already accessible via alliancecan#hpss Globus endpoint. Directory tree now follows the other Alliance clusters. We're still working on job submission via Slurm

'''Thu Oct 01, 2025, 0:00 am:''' Niagara compute nodes are now unavailable for regular users. The login nodes will remain available for a while to allow a few last data transfers, although transfers from the Niagara file systems to Trillium are best done on nia-dm1.scinet.utoronto.ca.

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-01-16T03:08:19Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up | Globus |Globus}}
|-
|{{Down | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Thu Jan 15, 2026, 10:00 pm:''' HPSS will undergo maintenance on Friday morning, Jan/16/2025, , including alliancecan#hpss Globus endpoint

'''Tue Jan 6, 2026, 10:15 am:''' OnDemand has been fixed and is working again.

'''Mon Jan 5, 2026, 9:00 pm:''' The authentication mechanism of OnDemand is not working.

'''Wed Dec 31, 2025, 12:40 pm:''' We believe the problem has now been resolved. Please let us know if you still experience login problems or aborted jobs.

'''Tue Dec 30, 2025, 2:10 pm:''' We are experiencing problems with authentication, resulting in failed logins, OOD errors, and aborted jobs (with "prolog error"). Please bear with us, as we are very short-staffed during the holiday break. We will post updates here.

'''Tue Dec 3, 2025, 11:30 am:''' Open OnDemand is fully operational again.

'''Sat Nov 29, 2025, 00:40 am:''' There has been a problem with the water chiller. Some systems are offline.

'''Wed Nov 5, 2025, 12:55 pm:''' Balam is back online.

'''Wed Nov 5, 2025, 10:00 am:''' Open OnDemand is back online.

'''Tue Nov 4, 2025, 11:00 pm:''' Most of the work is done, data movers, Globus, and HPSS are back online. Remaining services will be worked on tomorrow.

'''Tue Nov 4, 2025, 8:30 am:''' Scheduled network maintenance. Trillium cluster is *not* affected.

'''Tue Oct 21, 2025, 17:30 am:''' Balam maintenance finished.

'''Tue Oct 21, 2025, 7:00 am:''' Balam maintenance day.

'''Thu Oct 15, 2025, 3:55 pm:''' Trillium inbound connections through trillium.alliancecan.ca or trillium.scinet.utoronto.ca are working again.

'''Thu Oct 15, 2025, 3:05 pm:''' Trillium is experiencing external network issues for both incoming traffic. Please try: ssh USERNAME@tri-login01.scinet.utoronto.ca in the meantime.

'''Thu Oct 06, 2025, 8:00 pm:''' HPSS is fully functional. You may submit archive jobs from trillium login nodes, datamovers and robots.

'''Thu Oct 03, 2025, 6:30 pm:''' HPSS is back online, and already accessible via alliancecan#hpss Globus endpoint. Directory tree now follows the other Alliance clusters. We're still working on job submission via Slurm

'''Thu Oct 01, 2025, 0:00 am:''' Niagara compute nodes are now unavailable for regular users. The login nodes will remain available for a while to allow a few last data transfers, although transfers from the Niagara file systems to Trillium are best done on nia-dm1.scinet.utoronto.ca.

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2026-01-16T03:07:26Z

Pinto:

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up3 | OnDemand|https://docs.alliancecan.ca/wiki/Trillium_Open_OnDemand_Quickstart}}
|{{Up | Globus |Globus}}
|-
|{{Down | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Up | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Thu Jan 15, 2026, 10:00 pm:''' HPSS will undergo maintenance on Friday morning, Jan/16/2025

'''Tue Jan 6, 2026, 10:15 am:''' OnDemand has been fixed and is working again.

'''Mon Jan 5, 2026, 9:00 pm:''' The authentication mechanism of OnDemand is not working.

'''Wed Dec 31, 2025, 12:40 pm:''' We believe the problem has now been resolved. Please let us know if you still experience login problems or aborted jobs.

'''Tue Dec 30, 2025, 2:10 pm:''' We are experiencing problems with authentication, resulting in failed logins, OOD errors, and aborted jobs (with "prolog error"). Please bear with us, as we are very short-staffed during the holiday break. We will post updates here.

'''Tue Dec 3, 2025, 11:30 am:''' Open OnDemand is fully operational again.

'''Sat Nov 29, 2025, 00:40 am:''' There has been a problem with the water chiller. Some systems are offline.

'''Wed Nov 5, 2025, 12:55 pm:''' Balam is back online.

'''Wed Nov 5, 2025, 10:00 am:''' Open OnDemand is back online.

'''Tue Nov 4, 2025, 11:00 pm:''' Most of the work is done, data movers, Globus, and HPSS are back online. Remaining services will be worked on tomorrow.

'''Tue Nov 4, 2025, 8:30 am:''' Scheduled network maintenance. Trillium cluster is *not* affected.

'''Tue Oct 21, 2025, 17:30 am:''' Balam maintenance finished.

'''Tue Oct 21, 2025, 7:00 am:''' Balam maintenance day.

'''Thu Oct 15, 2025, 3:55 pm:''' Trillium inbound connections through trillium.alliancecan.ca or trillium.scinet.utoronto.ca are working again.

'''Thu Oct 15, 2025, 3:05 pm:''' Trillium is experiencing external network issues for both incoming traffic. Please try: ssh USERNAME@tri-login01.scinet.utoronto.ca in the meantime.

'''Thu Oct 06, 2025, 8:00 pm:''' HPSS is fully functional. You may submit archive jobs from trillium login nodes, datamovers and robots.

'''Thu Oct 03, 2025, 6:30 pm:''' HPSS is back online, and already accessible via alliancecan#hpss Globus endpoint. Directory tree now follows the other Alliance clusters. We're still working on job submission via Slurm

'''Thu Oct 01, 2025, 0:00 am:''' Niagara compute nodes are now unavailable for regular users. The login nodes will remain available for a while to allow a few last data transfers, although transfers from the Niagara file systems to Trillium are best done on nia-dm1.scinet.utoronto.ca.

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

HPSS

2025-12-24T00:28:13Z

Pinto: /* HTAR */

{|align=right
|align=center|'''Topology Overview'''
|-
|[[Image:HPSS-overview.jpg|right|x600px]]
|-
|}

== High Performance Storage System ==

The High Performance Storage System ([https://www.hpss-collaboration.org/index.shtml HPSS] [https://en.wikipedia.org/wiki/High_Performance_Storage_System wikipedia]) is a tape-backed hierarchical storage system that provides a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data can be returned to the active shared parallel filesystem on Trillium ("VAST") when it is needed.

In the context of the {{Alliance}} Resource Allocations, it is called 'nearline'.

Since this system is intended for large data storage, it is accessible only to groups who have been awarded storage space at SciNet beyond 5TB in the yearly RAC resource allocation round. However, upon request, any user may be awarded access to HPSS, up to 2TB per group, so that you may get familiar with the system (just email support@scinet.utoronto.ca)

Access and transfer of data into and out of HPSS is done under the control of the user, whose interaction is expected to be scripted and submitted as a batch job, using one or more of the following utilities:
* [https://www.racf.bnl.gov/Facility/HPSS/Documentation/HSI/doc_intro.html HSI] is a client with an ftp-like functionality which can be used to archive and retrieve large files. It is also useful for browsing the contents of HPSS.
* [https://www.sdcc.bnl.gov/sites/default/files/2021-09/htar.txt HTAR] is a utility that creates tar formatted archives directly into HPSS. It also creates a separate index file (.idx) that can be accessed and browsed quickly.
* [https://support.scinet.utoronto.ca/wiki/index.php/ISH ISH] is a TUI utility that can perform an inventory of the files and directories in your tarballs.

We're currently running HPSS v 11.2.

== Why should I use and trust HPSS? ==
* HPSS is a 25 year-old collaboration between IBM and the DoE labs in the US, and is used by about 45 facilities in the [http://www.top500.org “Top 500”] HPC list (plus some black-sites).
* Over 2.5 ExaBytes of combined storage world-wide.
* The top 3 sites in the World report (fall 2017) having 360PB, 220PB and 125PB in production (ECMWF, UKMO and BNL)
* Environment Canada also adopted HPSS in 2017 to store Nav Canada data as well as to serve as their own archive. Currently has 2 X 100PB capacity installed.
* The SciNet HPSS system has been providing nearline capacity for important research data in Canada since early 2011, already at 10PB levels in 2018
* Very reliable, data redundancy and data insurance built-in (dual copies of everything are kept on tapes at SciNet)
* Data on cache and tapes can be geo-distributed for further resilience and HA.
* Highly scalable; current performance at SciNet - after the hardware and software upgrades in 2019 - Ingest and Recalls ~150 TB/day aggregated.
* HSI/HTAR clients also very reliable and used on several HPSS sites. ISH was written at SciNet.
* [[Media:HPSS_rationale_SNUG.pdf|HPSS fits well with the Storage Capacity Expansion Plan at SciNet]] (pdf presentation)

== Guidelines ==
* A large portion of the storage capacity of HPSS is provided on tape -- a media that is not suited for storing small files. Files smaller than ~200MB should be grouped into tarballs with '''tar''' or '''htar'''.
* If you intent to use '''HSI''' or '''Globus''' to ingest material, you should favor tarballs over expanded directory trees. Your average file size on archive should be over 1GB/file.
* Optimal performance for aggregated transfers and allocation on tapes is obtained with [[Why not tarballs too large |tarballs of size 1TB or less]], whether ingested by htar or hsi ([[Why not tarballs too large |for good reasons]])
* We strongly urge that you use the sample scripts we are providing as the basis for your job submissions.
* Make sure to check the exit codes of each of the steps in your scripts, and check any returned logs for errors after any data transfer or tarball creation process.

== '''#### New to the HPSS/nearline/archive System? ####''' ==

First, nearline, archive, and HPSS all mean the same thing at SciNet. HPSS is the true name of this storage system.

Second, HPSS is not a regular file system, and as such is not mounted on the regular login, compute, and datamover nodes. Interaction with the HPSS system is therefore different than using simple copy, move, and tar commands.

Before trying to use the system, you need to confirm you are part of a group that already has a HPSS RAC allocation, otherwise, email SciNet support and request an HPSS account (or else you will get "Error - authentication/initialization failed" and 71 exit codes).

THIS set of instructions on the wiki is the best and most compressed "manual" we have on how to use HPSS. It may seem a bit overwhelming at first, because of all the job script templates we make available below (they are here so you don't have to think
too much, just copy and paste), but if you approach the index at the top as a "case switch" mechanism for what you intend to do, everything falls in place.

== The 3 ways to access HPSS ==
* Submitting jobs to the [[HPSS#Access_Through_the_Queue_System | Trillium archive partitions]] (archiveshort or archivelong), and using HTAR or HSI on the archive02 node. These tools are optimized for speed and scale, to efficiently deal with small files, and are very resilient to timeouts due to slow tape access. That is our recommendation for large transfers.
* Using Globus, which can be a very good trade off between the efficiency of HSI/HTAR and the convenience of a WebGUI. You may transfer data between 2 endpoints. For instance:
alliancecan#trillium
alliancecan#hpss
* Submitting an interactive job to the [[HPSS#Access_Through_the_Queue_System | VFS partition]] (vfsshort), and using standard linux tools (cp, rsync, ls, cd, ...) to access the VFS based /archive mount point on the vfs02 node. We don't recommend this way, unless you are transferring only a handful of files, or just navigating the naming-space (directory tree similar to the other Trillium file systems).
/archive

Try this sequence:

1) [https://docs.scinet.utoronto.ca/index.php/HPSS#Access_Through_an_Interactive_HSI_session take a look around HPSS using an interactive HSI session]

(most linux shell commands have an equivalent in HPSS)

2) [https://docs.scinet.utoronto.ca/index.php/HPSS#Sample_tarball_create archive a small test directory using HTAR]

2a) use step 1) to see what happened

3) [https://docs.scinet.utoronto.ca/index.php/HPSS#Sample_data_offload archive a file using hsi]

3a) use step 1) to see what happened

4) [https://docs.scinet.utoronto.ca/index.php/HPSS#Sample_transferring_directories archive a small test directory using HSI]

4a) use step 1) to see what happened

5) now try the other cases and so on. In a couple of hours you'll be in pretty good shape.

== Access Through the Queue System ==
All access to the archive system is done through the [https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Submitting_jobs_to_the_scheduler Trillium queue system].

* Job submissions should be done to the 'archivelong', 'archiveshort' or 'vfsshort' partitions.
* Short jobs are limited to 1H walltime by default. Long jobs (> 1H) are limited to 72H walltime.
* Users are limited to only 2 long jobs and 2 short jobs at the same time, and 10 jobs total on the each queue.
* There can only be 5 long jobs running at any given time overall. Remaining submissions will be placed on hold for the time being. So far we have not seen a need for overall limit on short jobs.
* Jobs to the 'archivelong' and 'archiveshort' partitions must use [[HPSS#HTAR | htar]] or [[HPSS#HSI | hsi]]. These tools are optimized for speed and scale, to efficiently deal with small files, and are very resilient to timeouts due to slow tape access.
* The 'vfsshort' partition can only be used for interactive jobs, so that you can navigate the VFS based /archive mount point (Virtual File System), and/or make small file transfers to/from HPSS using standard linux tools (ls, cd, cp, rsync, etc). This access is always limited to 1 hour.

You can submit your job like this:
<pre>
sbatch jobscript.sh (to 'archivelong' or 'archiveshort')
</pre>

And you can get an interactive session like this:
<pre>
salloc -p archiveshort
OR
salloc -p vfsshort
</pre>

[https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Submitting_jobs_to_the_scheduler You may learn SLURM in more details here.]

The status of pending jobs can be monitored with squeue specifying the archive partition:
<pre>
squeue -p archiveshort
OR
squeue -p archivelong
OR
squeue -p vfsshort
</pre>

== Access Through an Interactive HSI session ==
* You may want to acquire an interactive shell, start an HSI session and navigate the archive naming-space. Keep in mind, you're restricted to 1H.

<pre>
pinto@tri-login02:~$ salloc -p archiveshort -t 1:00:00
salloc: Granted job allocation 50918
salloc: Waiting for resource configuration
salloc: Nodes hpss-archive02-ib are ready for job
hpss-archive02-ib:~$

hpss-archive02-ib:~$ hsi (DON'T FORGET TO START HSI)
******************************************************************
* Welcome to HPSS@SciNet - High Perfomance Storage System *
* *
* INFO: THIS IS THE NEW 7.5.1 HPSS SYSTEM! *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
* CHECK THE INTEGRITY OF YOUR TARBALLS *
******************************************************************
[HSI]/archive/scinet/pinto-> ls

[HSI]/archive/scinet/pinto-> cd <some directory>
</pre>

NOTE: the VFS based /archive mount point is not visible on the archive02 node with the standard linux prompt. You must use HSI (or get an interactive session on the vfsshort partition).

=== Scripted File Transfers ===
File transfers in and out of the HPSS should be scripted into jobs and submitted to the ''archivelong'' partition or the ''archiveshort'' partition. See generic example below:

<source lang="bash">
#!/bin/bash -l
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J htar_create_tarball_in_hpss
#SBATCH --mail-type=ALL

echo "Creating a htar of finished-job1/ directory tree into HPSS"
trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

DEST=$ARCHIVE/finished-job1.tar

# htar WILL overwrite an existing file with the same name so check beforehand.

hsi ls $DEST &> /dev/null
status=$?

if [ $status == 0 ]; then
echo 'File $DEST already exists. Nothing has been done'
exit 1
fi

cd $SCRATCH/workarea/
htar -Humask=0137 -cpf $ARCHIVE/finished-job1.tar finished-job1/
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>
'''Note:''' Always trap the execution of your jobs for abnormal terminations, and be sure to return the exit code

=== Job Dependencies ===

Typically data will be recalled to /scratch when it is needed for analysis. Job dependencies can be constructed so that analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
--dependency=<type:JOBID>
</pre>
where JOBID is the job number of the archive recalling job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency (lookup [https://docs.scinet.utoronto.ca/index.php/HPSS#Sample_data_recall data-recall.sh samples]):
<pre>
hpss-archive02-ib:~$ sbatch -d afterok:$(sbatch --parsable data-recall.sh) job-to-work-on-recalled-data.sh
</pre>

== HTAR ==
''' Please aggregate small files (<~200MB) into tarballs or htar files. '''

''' [[Why not tarballs too large |Keep your tarballs to size 1TB or less]], whether ingested by htar or hsi ([[Why not tarballs too large | WHY?]])'''

HTAR is a utility that is used for aggregating a set of files and directories, by using a sophisticated multithreaded buffering scheme to write files directly from VAST into HPSS, creating an archive file that conforms to the POSIX TAR specification, thereby achieving a high rate of performance. HTAR does not do gzip compression, however it already has a built-in checksum algorithm.

'''Caution'''
* Unlike with cput/cget in HSI, "prompt before overwrite", this is not the default with (h)tar. Be careful not to unintentionally overwrite a previous htar destination file in HPSS. There could be a similar situation when extracting material back into VAST and overwriting the originals. Be sure to double-check the logic in your scripts.
* Check the HTAR exit code and log file before removing any files from the VAST active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, and preserve mask attributes (-p), enter:
<pre>
htar -cpf files.tar file1 file2
OR
htar -cpf $ARCHIVE/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cpf subdirA.tar subdirA/
</pre>

* To extract all files from the archive file called ''proj1.tar'' in HPSS into the ''project1/src'' directory in VAST, and use the time of extraction as the modification time, enter:
<pre>
cd project1/src
htar -xpmf proj1.tar
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

* To ensure that both the htar and the .idx files have read permissions to other members in your group use the umask option
<pre>
htar -Humask=0137 ....
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

==== Sample tarball create ====
<source lang="bash">
#!/bin/bash -l
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J htar_create_tarball_in_hpss
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

DEST=$ARCHIVE/finished-job1.tar

# htar WILL overwrite an existing file with the same name so check beforehand.

hsi ls $DEST &> /dev/null
status=$?

if [ $status == 0 ]; then
echo 'File $DEST already exists. Nothing has been done'
exit 1
fi

cd $SCRATCH/workarea/
htar -Humask=0137 -cpf $DEST finished-job1/
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

'''Note:''' If you attempt to start a transfer with any files larger than 68GB the whole HTAR session will fail, and you'll get a notification listing all those files, so that you can transfer them with HSI.
<pre>
----------------------------------------
INFO: File too large for htar to handle: finished-job1/file1 (86567185745 bytes)
INFO: File too large for htar to handle: finished-job1/file2 (71857244579 bytes)
ERROR: 2 oversize member files found - please correct and retry
ERROR: [FATAL] error(s) generating filename list
HTAR: HTAR FAILED
###WARNING htar returned non-zero exit status
</pre>

==== Sample tarball list ====
<source lang="bash">
#!/bin/bash -l
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J htar_list_tarball_in_hpss
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

DEST=$ARCHIVE/finished-job1.tar

htar -tvf $DEST
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Sample tarball extract ====
<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J htar_extract_tarball_from_hpss
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

cd $SCRATCH/recalled-from-hpss
htar -xpmf $ARCHIVE/finished-job1.tar
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

== HSI ==

HSI may be the primary client with which some users will interact with HPSS. It provides an ftp-like interface for archiving and retrieving tarballs or [https://docs.scinet.utoronto.ca/index.php/HPSS#Sample_transferring_directories directory trees]. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents in HPSS. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally saves or replaces a HPSSpath file to VASTpath if the VAST version is new or has been updated
cput [options] VASTpath [: HPSSpath]
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to VAST only if a VAST version does not already exist.
cget [options] [VASTpath :] HPSSpath
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of HPSS.
|-
| lcd,lls
| ''Local'' commands to VAST
|}

*There are 3 distinctions about HSI that you should keep in mind, and that can generate a bit of confusion when you're first learning how to use it:
** HSI doesn't currently support renaming directories paths during transfers on-the-fly, therefore the syntax for cput/cget may not work as one would expect in some scenarios, requiring some workarounds.
** HSI has an operator ":" which separates the VASTpath and HPSSpath, and must be surrounded by whitespace (one or more space characters)
** The order for referring to files in HSI syntax is different from FTP. In HSI the general format is always the same, VAST first, HPSS second, cput or cget:
<pre>
VASTfile : HPSSfile
</pre>
For example, when using HSI to store the tarball file from VAST into HPSS, then recall it to VAST, the following commands could be used:
<pre>
cput tarball-in-VAST : tarball-in-HPSS
cget tarball-recalled : tarball-in-HPSS
</pre>

unlike with FTP, where the following syntax would be used:
<pre>
put tarball-in-VAST tarball-in-HPSS
get tarball-in-HPSS tarball-recalled
</pre>
* Simple commands can be executed on a single line.
<pre>
hsi "mkdir LargeFilesDir; cd LargeFilesDir; cput tarball-in-VAST : tarball-in-HPSS"
</pre>

* More complex sequences can be performed using an except such as this:
<pre>
hsi <<EOF
mkdir LargeFilesDir
cd LargeFilesDir
cput tarball-in-VAST : tarball-in-HPSS
lcd $SCRATCH/LargeFilesDir2/
cput -Ruph *
end
EOF
</pre>

* The commands below are equivalent, but we recommend that you always use full path, and organize the contents of HPSS, where the default HSI directory placement is $ARCHIVE:
<pre>
hsi cput tarball
hsi cput tarball : tarball
hsi cput $SCRATCH/tarball : $ARCHIVE/tarball
</pre>

* There are no known issues renaming files on-the-fly:
<pre>
hsi cput $SCRATCH/tarball1 : $ARCHIVE/tarball2
hsi cget $SCRATCH/tarball3 : $ARCHIVE/tarball2
</pre>

* However the syntax forms such as the ones below will fail, since they rename the directory paths.
<pre>
hsi cput -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir (FAILS)
OR
hsi cget -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir2 (FAILS)
OR
hsi cput -Ruph $SCRATCH/LargeFilesDir/* : $ARCHIVE/LargeFilesDir2 (FAILS)
OR
hsi cget -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir (FAILS)
</pre>

One workaround is the following 2-steps process, where you do a "lcd " in VAST first, and recursively transfer the whole directory (-R), keeping the same name. You may use '-u' option to resume a previously disrupted session, and the '-p' to preserve timestamp, and '-h' to keep the links.
<pre>
hsi <<EOF
lcd $SCRATCH
cget -Ruph LargeFilesDir
end
EOF
</pre>

Another workaround is do a "lcd" into the VASTpath first and a "cd" in the HPSSpath, but transfer the files individually with the '*' wild character. This option lets you change the directory name:
<pre>
hsi <<EOF
lcd $SCRATCH/LargeFilesDir
mkdir $ARCHIVE/LargeFilesDir2
cd $ARCHIVE/LargeFilesDir2
cput -Ruph *
end
EOF
</pre>

=== Documentation ===
Complete documentation on HSI is available from the Gleicher Enterprises links below. You may peruse those links and come with alternative syntax forms. You may even be already familiar with HPSS/HSI from other HPC facilities, that may or not have procedures similar to ours. HSI doesn't always work as expected when you go outside of our recommended syntax, so '''we strongly urge that you use the sample scripts we are providing as the basis''' for your job submissions
* [http://www.mgleicher.us/index.html/hsi/hsi_reference_manual_2/introduction.html HSI Introduction] (original site inactive)
* [http://www.mgleicher.us/index.html/hsi/hsi_man_page.html man hsi] (original site inactive)
* [https://docs.scinet.utoronto.ca/index.php/HSI_help hsi help]
* [http://www.mgleicher.us/index.html/hsi/hsi-exit-codes.html exit codes] (original site inactive)
'''Note:''' HSI returns the highest-numbered exit code, in case of multiple operations in the same hsi session. You may use '/scinet/hpss/bin/exit2msg $status' to translate those codes into intelligible messages

=== Typical Usage Scripts===
The most common interactions will be ''putting'' data into HPSS, examining the contents (ls,ish), and ''getting'' data back onto VAST for inspection or analysis.

==== Sample data offload ====
<source lang="bash">
#!/bin/bash
# This script is named: data-offload.sh
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J offload
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# individual tarballs already exist

/usr/local/bin/hsi -v <<EOF1
mkdir put-away
cd put-away
cput $SCRATCH/workarea/finished-job1.tar.gz : finished-job1.tar.gz
end
EOF1
status=$?
if [ ! $status == 0 ];then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

/usr/local/bin/hsi -v <<EOF2
mkdir put-away
cd put-away
cput $SCRATCH/workarea/finished-job2.tar.gz : finished-job2.tar.gz
end
EOF2
status=$?
if [ ! $status == 0 ];then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

trap - TERM INT
</source>

'''Note:''' as in the above example, we recommend that you capture the (highest-numbered) exit code for each hsi session independently. And remember, you may improve your exit code verbosity by adding the excerpt below to your scripts:
<source lang="bash">
if [ ! $status == 0 ];then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Sample data list ====
A very trivial way to list the contents of HPSS would be to just submit the HSI 'ls' command.
<source lang="bash">
#!/bin/bash
# This script is named: data-list.sh
#SBATCH -t 1:00:00
#SBATCH -p archiveshort
#SBATCH -N 1
#SBATCH -J hpss_ls
#SBATCH --mail-type=ALL

/usr/local/bin/hsi -v <<EOF
cd put-away
ls -R
end
EOF
</source>
''Warning: if you have a lot of files, the ls command will take a long time to complete. For instance, about 400,000 files can be listed in about an hour. Adjust the walltime accordingly, and be on the safe side.''

However, we provide a much more useful and convenient way to explore the contents of HPSS with the inventory shell [[ISH]]. This example creates an index of all the files in a user's portion of the namespace. The list is placed in the directory /home/$(whoami)/.ish_register that can be inspected from the login nodes.
<source lang="bash">
#!/bin/bash
# This script is named: data-list.sh
#SBATCH -t 1:00:00
#SBATCH -p archiveshort
#SBATCH -N 1
#SBATCH -J hpss_index
#SBATCH --mail-type=ALL

INDEX_DIR=$HOME/.ish_register
if ! [ -e "$INDEX_DIR" ]; then
mkdir -p $INDEX_DIR
fi

export ISHREGISTER="$INDEX_DIR"
/scinet/hpss/bin/ish hindex
</source>
''Note: the above warning on collecting the listing for many files applies here too.''

This index can be browsed or searched with ISH on the development nodes.
<source lang="bash">
hpss-archive02-ib:~$ /scinet/hpss/bin/ish ~/.ish_register/hpss.igz
[ish]hpss.igz> help
</source>

ISH is a powerful tool that is also useful for creating and browsing indices of tar and htar archives, so please look at the [[ISH|documentation]] or built in help.

==== Sample data recall ====
<source lang="bash">
#!/bin/bash
# This script is named: data-recall.sh
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J recall_files
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
mkdir -p $SCRATCH/recalled-from-hpss

# individual tarballs previously organized in HPSS inside the put-away-on-2010/ folder
hsi -v << EOF
cget $SCRATCH/recalled-from-hpss/Jan-2010-jobs.tar.gz : $ARCHIVE/put-away-on-2010/Jan-2010-jobs.tar.gz
cget $SCRATCH/recalled-from-hpss/Feb-2010-jobs.tar.gz : $ARCHIVE/put-away-on-2010/Feb-2010-jobs.tar.gz
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

We should emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do optimization, as in the following example:
<source lang="bash">
#!/bin/bash
# This script is named: data-recall.sh
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J recall_files_optimized
#SBATCH --mail-type=AL

trap "echo 'Job script not completed';exit 129" TERM INT
mkdir -p $SCRATCH/recalled-from-hpss

# individual tarballs previously organized in HPSS inside the put-away-on-2010/ folder
hsi -v << EOF
lcd $SCRATCH/recalled-from-hpss/
cd $ARCHIVE/put-away-on-2010/
cget Jan-2010-jobs.tar.gz Feb-2010-jobs.tar.gz
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

=== Sample transferring directories ===
Remember, it's not possible to rename directories or paths on-the-fly:
<pre>
hsi cget -Ruph $SCRATCH/LargeFiles-recalled : $ARCHIVE/LargeFiles (FAILS)
</pre>

One workaround is transfer the whole directory (and sub-directories) recursively:
<source lang="bash">
#!/bin/bash
# This script is named: data-recall.sh
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J recall_directories
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
mkdir -p $SCRATCH/recalled

hsi -v << EOF
lcd $SCRATCH/recalled
cd $ARCHIVE/
cget -Ruph LargeFiles
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

Another workaround is to transfer files and subdirectories individually with the "*" wild character:
<source lang="bash">
#!/bin/bash
# This script is named: data-recall.sh
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J recall_directories
#SBATCH --mail-type=AL

trap "echo 'Job script not completed';exit 129" TERM INT
mkdir -p $SCRATCH/LargeFiles-recalled

hsi -v << EOF
lcd $SCRATCH/LargeFiles-recalled
cd $ARCHIVE/LargeFiles
cget -Ruph *
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help''']

== [[ISH|ISH]] ==
=== [[ISH|Documentation and Usage]] ===

== File and directory management ==
=== Moving/renaming ===
* you may use 'mv' or 'cp' in the same way as the linux version.
<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J deletion_script
#SBATCH --mail-type=ALL

echo "HPSS file and directory management"
trap "echo 'Job script not completed';exit 129" TERM INT

/usr/local/bin/hsi -v <<EOF1
mkdir $ARCHIVE/2011
mv $ARCHIVE/oldjobs $ARCHIVE/2011
cp -r $ARCHIVE/almostfinished/*done $ARCHIVE/2011
end
EOF1
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

=== Deletions ===
==== Recommendations ====
* Be careful with the use of 'cd' commands to non-existing directories before the 'rm' command. Results may be unpredictable
* Avoid the use of the stand alone wild character '''*'''. If necessary, whenever possible have it bound to common patterns, such as '*.tmp', so to limit unintentional mis-happens
* Avoid using relative paths, even the env variable $ARCHIVE. Better to explicitly expand the full paths in your scripts
* Avoid using recursive/looped deletion instructions on $SCRATCH contents from the archive job scripts. Even on $ARCHIVE contents, it may be better to do it as an independent job submission, after you have verified that the original ingestion into HPSS finished without any issues.

==== Typical example ====
<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J deletion_script
#SBATCH --mail-type=ALL

echo "Deletion of an outdated directory tree into HPSS"

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

/usr/local/bin/hsi -v <<EOF1
rm /archive/scinet/pinto/*.tmp
rm -R /archive/scinet/pinto/obsolete
end
EOF1
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Deleting with an interactive HSI session ====
* You may feel more comfortable acquiring an interactive shell, starting an HSI session and proceeding with your deletions that way. Keep in mind, you're restricted to 1H.

* After using the ''sbatch -p archiveshort'' command you'll get a standard shell prompt on an archive execution node (hpss-archive02), as you would on any compute node. However you will need to run '''HSI''' or '''HTAR''' to access resources on HPSS.

* HSI will give you a prompt very similar to a standard shell, where your can navigate around using commands such 'ls', 'cd', 'pwd', etc ... NOTE: not every bash command has an equivalent on HSI - for instance, you can not 'vi' or 'cat'.

<pre>
pinto@nia-login07:~$ salloc -p archiveshort -t 1:00:00
salloc: Granted job allocation 50359
salloc: Waiting for resource configuration
salloc: Nodes hpss-archive02-ib are ready for job

hpss-archive02-ib:~$ hsi
******************************************************************
* Welcome to HPSS@SciNet - High Perfomance Storage System *
* *
* INFO: THIS IS THE NEW 7.5.1 HPSS SYSTEM! *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
* CHECK THE INTEGRITY OF YOUR TARBALLS *
******************************************************************

[HSI]/archive/scinet/pinto-> rm -R junk

</pre>

== HPSS for the 'Watchmaker' ==
=== Efficient alternative to htar ===
<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J tar_create_tarball_in_hpss_with_hsi_by_piping
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

# When using a pipeline like this
set -o pipefail

# to put (cput will fail)
tar -c $SCRATCH/mydir | hsi put - : $ARCHIVE/mydir.tar
status=$?
if [ ! $status == 0 ]; then
echo 'TAR+HSI+piping returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# to immediately generate an index
ish hindex $ARCHIVE/mydir.tar
status=$?
if [ ! $status == 0 ]; then
echo 'ISH returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# to get
#cd $SCRATCH
#hsi cget - : $ARCHIVE/mydir.tar | tar -xv
#status=$?
# if [ ! $status == 0 ]; then
# echo 'TAR+HSI+piping returned non-zero code.'
# /scinet/hpss/bin/exit2msg $status
# exit $status
#else
# echo 'TRANSFER SUCCESSFUL'
#fi

trap - TERM INT

</source>
'''Notes:'''
* Combining commands in this fashion, besides being HPSS-friendly, should not be that noticeably slower than the recursive put with HSI that stores each file one by one. However, reading the files back from tape in this format will be many times faster. It would also overcome the current 68GB limit on the size of stored files that we have with htar.
* To top things off, we recommend indexing with ish (in the same script) immediately after the tarball creation , while it resides in the HPSS cache. It would be as if htar was used.
* To ensure that an error at any stage of the pipeline shows up in the returned status use: ''set -o pipefail'' (The default is to return the status of the last command in the pipeline and this is not what you want.)
* Optimal performance for aggregated transfers and allocation on tapes is obtained with [[Why not tarballs too large |tarballs of size 500GB or less]], whether ingested by htar or hsi ([[Why not tarballs too large | WHY?]]). Be sure to check the contents of the directory tree with 'du' for the total amount of data before sending them to the tar+HSI piping.

=== Multi-threaded gzip'ed compression with pigz ===
We compiled multi-threaded implementation of gzip called pigz (http://zlib.net/pigz/). It's now part of the "extras" module. It can also be used on any compute or devel nodes. This makes the execution of the previous version of the script much quicker than if you were to use 'tar -cfz'. In addition, by piggy-backing ISH to the end of the script, it will know what to do with the just created mydir.tar.gz compressed tarball.

<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J tar_create_compressed_tarball_in_hpss_with_hsi_by_piping
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

# When using a pipeline like this
set -o pipefail

# to put (cput will fail)
tar -c $SCRATCH/mydir | pigz | hsi put - : $ARCHIVE/mydir.tar.gz
status=$?
if [ ! $status == 0 ]; then
echo 'TAR+PIGZ+HSI+piping returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

NOTE: Do not use this type of pipeline to serialize recalls with HSI inside a for loop. This is a very inefficient way for HPSS to handle this type of operation, since it forces the system to fetch the tarballs using your sequence on the loop, or a sequence based on a numeric/alphabetic/chronological order from the tapes. Files are not necessarily written to tapes in the same sequence in which they are ingested. There will be many unnecessary mounts/dismounts involved, and a lot of fast forward and rewinds, depending on where the files are on the tapes, which can be extremely stressful to the hardware. This effect is know a "shoe-shinning".

Instead, provide HSI with a full list of tarballs to be recalled, all at once, and the HSI will sort the list in the most convenient order possible, in the sequence that the files are laid out on the tapes, and fetch all the files from each tape at once, in one go.
Afterwards, once all the tarballs are back onto your $SCRATCH, you may use a loop to unpigz the tarballs

=== Content Verification ===

==== HTAR CRC checksums ====
Specifies that HTAR should generate CRC checksums when creating the archive.

<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J htar_create_tarball_in_hpss_with_checksum_verification
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

cd $SCRATCH/workarea

# to put
htar -Humask=0137 -cpf $ARCHIVE/finished-job1.tar -Hcrc -Hverify=1 finished-job1/

# to get
#mkdir $SCRATCH/verification
#cd $SCRATCH/verification
#htar -Hcrc -xvpmf $ARCHIVE/finished-job1.tar

status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Current HSI version - Checksum built-in ====

MD5 is the standard Hashing Algorithm for the HSI build at SciNet. For hsi ingestions with the '-c on' option you should be able to query the md5 hash with the hsi command 'hashli'. That value is stored as an UDA (User Defined Attribute) for each file (a feature of HPSS starting with 7.4)

[http://www.mgleicher.us/GEL/hsi/hsi/hsi_reference_manual_2/checksum-feature.html More usage details here]

The checksum algorithm is very CPU-intensive. Although the checksum code is compiled with a high level of compiler optimization, transfer rates can be significantly reduced when checksum creation or verification is in effect. The amount of degradation in transfer rates depends on several factors, such as processor speed, network transfer speed, and speed of the local filesystem (VAST).

<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J MD5_checksum_verified_transfer
#SBATCH --mail-type=ALL

thefile=<VASTpath>
storedfile=<HPSSpath>

# Generate checksum on fly (-c on)
hsi -q put -c on $thefile : $storedfile
pid=$!

# Check the exit code of the HSI process
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# verify checksum
hsi lshash $storedfile
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# get the file back with checksum
hsi get -c on $storedfile
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Prior to HSI version 4.0.1.1 ====

This will checksum the contents of the HPSSpath against the original VASTpath after the transfer has finished.
<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J checksum_verified_transfer
#SBATCH --mail-type=ALL

thefile=<VASTpath>
storedfile=<HPSSpath>

# Generate checksum on fly using a named pipe so that file is only read from VAST once
mkfifo /tmp/NPIPE
cat $thefile | tee /tmp/NPIPE | hsi -q put - : $storedfile &
pid=$!
md5sum /tmp/NPIPE |tee /tmp/$fname.md5
rm -f /tmp/NPIPE

# Check the exit code of the HSI process
wait $pid
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# change filename to stdin in checksum file
sed -i.1 "s+/tmp/NPIPE+-+" /tmp/$fname.md5

# verify checksum
hsi -q get - : $storedfile | md5sum -c /tmp/$fname.md5
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

=== Splitting tarballs ===

By knowing the size of your files or directories you can decide how to divide and organize them in different archives if necessary, so not to create huge tarballs to start with.

However, if you find it more convenient to bundle a whole directory into a big tarball, you can still divide it into small chunks prior to ingesting them into HPSS, by using the following syntax:

<code>split -b <Size-in-GB> <tar-file-name> <prefix-name></code>

<source lang="console">
$ split -b 500GB results.tar small-chunk
</source>

The option '''b''' is invoked to fix the size of the small chunks; and '''prefix-name''' is the base name for the small chunks. The above command will split '''results.tar''' into 500 GB chunks in current working directory. The small chunks will starts from: small-chunka small-chunkb small-chunkc small-chunkd .... etc. You may then ingest them into HPSS using any of the 3 methods describe above (HSI, VFS, Globus).

To rebuild the original tarball, recall the individual chunks from HPSS and use the <code>cat</code> command as follows:

<source lang="console">
$ cat small-chunk* > your_archive_name.tar
</source>

In case you want to append numbers in place of alphabets, use '''-d''' option in above split command.

== Access to HPSS using Globus ==
* You may now transfer data between SciNet's HPSS and an external source
* Follow the link below
https://globus.alliancecan.ca
: Enter your {{Alliance}} username and password (and use your MFA).
* In the 'File Manager' tab, enter ''''alliancecan#hpss'''' as one of the Endpoints. If you are seeing 'Missing required data_access_consent', authenticate this endpoint by clicking on 'Continue', enter your username and password once more.
* You may read more about {{Alliance}}} Globus Portal here:
https://docs.alliancecan.ca/wiki/Globus

== User provided Content/Suggestions ==

== [[HPSS-by-pomes|Packing up large data sets and putting them on HPSS]] ==
(Pomés group recommendations)

[[Data Management|BACK TO Data Management]]

Main Page

2025-12-04T15:21:57Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Up | OnDemand|Open_OnDemand_Quickstart}}
|{{Up | Globus |Globus}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Balam|Balam}}
|{{Up | S4H | S4H}}
|-
|{{Down | Teach|Teach}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|}

'''Tue Dec 3, 2025, 11:30 am:''' Open OnDemand is fully operational again.

'''Sat Nov 29, 2025, 00:40 am:''' There has been a problem with the water chiller. Some systems are offline.

'''Wed Nov 5, 2025, 12:55 pm:''' Balam is back online.

'''Wed Nov 5, 2025, 10:00 am:''' Open OnDemand is back online.

'''Tue Nov 4, 2025, 11:00 pm:''' Most of the work is done, data movers, Globus, and HPSS are back online. Remaining services will be worked on tomorrow.

'''Tue Nov 4, 2025, 8:30 am:''' Scheduled network maintenance. Trillium cluster is *not* affected.

'''Tue Oct 21, 2025, 17:30 am:''' Balam maintenance finished.

'''Tue Oct 21, 2025, 7:00 am:''' Balam maintenance day.

'''Thu Oct 15, 2025, 3:55 pm:''' Trillium inbound connections through trillium.alliancecan.ca or trillium.scinet.utoronto.ca are working again.

'''Thu Oct 15, 2025, 3:05 pm:''' Trillium is experiencing external network issues for both incoming traffic. Please try: ssh USERNAME@tri-login01.scinet.utoronto.ca in the meantime.

'''Thu Oct 06, 2025, 8:00 pm:''' HPSS is fully functional. You may submit archive jobs from trillium login nodes, datamovers and robots.

'''Thu Oct 03, 2025, 6:30 pm:''' HPSS is back online, and already accessible via alliancecan#hpss Globus endpoint. Directory tree now follows the other Alliance clusters. We're still working on job submission via Slurm

'''Thu Oct 01, 2025, 0:00 am:''' Niagara compute nodes are now unavailable for regular users. The login nodes will remain available for a while to allow a few last data transfers, although transfers from the Niagara file systems to Trillium are best done on nia-dm1.scinet.utoronto.ca.

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2025-12-04T14:33:12Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Down | Niagara|Niagara_Quickstart}}
|{{Down | Teach|Teach}}
|{{Down | Rouge|Rouge}}
|-
|{{Up | OnDemand|Open_OnDemand_Quickstart}}
|{{Up3 | Scheduler|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Submitting_jobs_to_the_scheduler}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Down | HPSS|HPSS}}
|{{Up3 | Login Nodes|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|{{Up | Globus |Globus}}
|-
|{{Up | Balam|Balam}}
|{{Up3 | Cvmfs|https://docs.alliancecan.ca/wiki/Standard_software_environments}}
|{{Up | S4H | S4H}}
|}

'''Tue Dec 3, 2025, 11:30 am:''' Open OnDemand is fully operational again.

'''Sat Nov 29, 2025, 00:40 am:''' There has been a problem with the water chiller. Some systems are offline.

'''Wed Nov 5, 2025, 12:55 pm:''' Balam is back online.

'''Wed Nov 5, 2025, 10:00 am:''' Open OnDemand is back online.

'''Tue Nov 4, 2025, 11:00 pm:''' Most of the work is done, data movers, Globus, and HPSS are back online. Remaining services will be worked on tomorrow.

'''Tue Nov 4, 2025, 8:30 am:''' Scheduled network maintenance. Trillium cluster is *not* affected.

'''Tue Oct 21, 2025, 17:30 am:''' Balam maintenance finished.

'''Tue Oct 21, 2025, 7:00 am:''' Balam maintenance day.

'''Thu Oct 15, 2025, 3:55 pm:''' Trillium inbound connections through trillium.alliancecan.ca or trillium.scinet.utoronto.ca are working again.

'''Thu Oct 15, 2025, 3:05 pm:''' Trillium is experiencing external network issues for both incoming traffic. Please try: ssh USERNAME@tri-login01.scinet.utoronto.ca in the meantime.

'''Thu Oct 06, 2025, 8:00 pm:''' HPSS is fully functional. You may submit archive jobs from trillium login nodes, datamovers and robots.

'''Thu Oct 03, 2025, 6:30 pm:''' HPSS is back online, and already accessible via alliancecan#hpss Globus endpoint. Directory tree now follows the other Alliance clusters. We're still working on job submission via Slurm

'''Thu Oct 01, 2025, 0:00 am:''' Niagara compute nodes are now unavailable for regular users. The login nodes will remain available for a while to allow a few last data transfers, although transfers from the Niagara file systems to Trillium are best done on nia-dm1.scinet.utoronto.ca.

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2025-12-01T18:47:09Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Down | Niagara|Niagara_Quickstart}}
|{{Down | Teach|Teach}}
|{{Down | Rouge|Rouge}}
|-
|{{Down | OnDemand|Open_OnDemand_Quickstart}}
|{{Up3 | Scheduler|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Submitting_jobs_to_the_scheduler}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Up | HPSS|HPSS}}
|{{Up3 | Login Nodes|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|{{Up | Globus |Globus}}
|-
|{{Up | Balam|Balam}}
|{{Up3 | Cvmfs|https://docs.alliancecan.ca/wiki/Standard_software_environments}}
|}

'''Sat Nov 29, 2025, 00:40 am:''' There has been a problem with the water chiller. Some systems are offline.

'''Wed Nov 5, 2025, 12:55 pm:''' Balam is back online.

'''Wed Nov 5, 2025, 10:00 am:''' Open OnDemand is back online.

'''Tue Nov 4, 2025, 11:00 pm:''' Most of the work is done, data movers, Globus, and HPSS are back online. Remaining services will be worked on tomorrow.

'''Tue Nov 4, 2025, 8:30 am:''' Scheduled network maintenance. Trillium cluster is *not* affected.

'''Tue Oct 21, 2025, 17:30 am:''' Balam maintenance finished.

'''Tue Oct 21, 2025, 7:00 am:''' Balam maintenance day.

'''Thu Oct 15, 2025, 3:55 pm:''' Trillium inbound connections through trillium.alliancecan.ca or trillium.scinet.utoronto.ca are working again.

'''Thu Oct 15, 2025, 3:05 pm:''' Trillium is experiencing external network issues for both incoming traffic. Please try: ssh USERNAME@tri-login01.scinet.utoronto.ca in the meantime.

'''Thu Oct 06, 2025, 8:00 pm:''' HPSS is fully functional. You may submit archive jobs from trillium login nodes, datamovers and robots.

'''Thu Oct 03, 2025, 6:30 pm:''' HPSS is back online, and already accessible via alliancecan#hpss Globus endpoint. Directory tree now follows the other Alliance clusters. We're still working on job submission via Slurm

'''Thu Oct 01, 2025, 0:00 am:''' Niagara compute nodes are now unavailable for regular users. The login nodes will remain available for a while to allow a few last data transfers, although transfers from the Niagara file systems to Trillium are best done on nia-dm1.scinet.utoronto.ca.

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2025-12-01T15:47:47Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Down | Niagara|Niagara_Quickstart}}
|{{Down | Teach|Teach}}
|{{Down | Rouge|Rouge}}
|-
|{{Down | OnDemand|Open_OnDemand_Quickstart}}
|{{Up3 | Scheduler|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Submitting_jobs_to_the_scheduler}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Up | HPSS|HPSS}}
|{{Up3 | Login Nodes|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|{{Up | Globus |Globus}}
|-
|{{Down | Balam|Balam}}
|{{Up3 | Cvmfs|https://docs.alliancecan.ca/wiki/Standard_software_environments}}
|}

'''Sat Nov 29, 2025, 00:40 am:''' There has been a problem with the water chiller. Some systems are offline.

'''Wed Nov 5, 2025, 12:55 pm:''' Balam is back online.

'''Wed Nov 5, 2025, 10:00 am:''' Open OnDemand is back online.

'''Tue Nov 4, 2025, 11:00 pm:''' Most of the work is done, data movers, Globus, and HPSS are back online. Remaining services will be worked on tomorrow.

'''Tue Nov 4, 2025, 8:30 am:''' Scheduled network maintenance. Trillium cluster is *not* affected.

'''Tue Oct 21, 2025, 17:30 am:''' Balam maintenance finished.

'''Tue Oct 21, 2025, 7:00 am:''' Balam maintenance day.

'''Thu Oct 15, 2025, 3:55 pm:''' Trillium inbound connections through trillium.alliancecan.ca or trillium.scinet.utoronto.ca are working again.

'''Thu Oct 15, 2025, 3:05 pm:''' Trillium is experiencing external network issues for both incoming traffic. Please try: ssh USERNAME@tri-login01.scinet.utoronto.ca in the meantime.

'''Thu Oct 06, 2025, 8:00 pm:''' HPSS is fully functional. You may submit archive jobs from trillium login nodes, datamovers and robots.

'''Thu Oct 03, 2025, 6:30 pm:''' HPSS is back online, and already accessible via alliancecan#hpss Globus endpoint. Directory tree now follows the other Alliance clusters. We're still working on job submission via Slurm

'''Thu Oct 01, 2025, 0:00 am:''' Niagara compute nodes are now unavailable for regular users. The login nodes will remain available for a while to allow a few last data transfers, although transfers from the Niagara file systems to Trillium are best done on nia-dm1.scinet.utoronto.ca.

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2025-12-01T15:47:26Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Down | Niagara|Niagara_Quickstart}}
|{{Down | Teach|Teach}}
|{{Up | Rouge|Rouge}}
|-
|{{Down | OnDemand|Open_OnDemand_Quickstart}}
|{{Up3 | Scheduler|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Submitting_jobs_to_the_scheduler}}
|{{Up3 | File system|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Storage}}
|{{Down | Burst Buffer|Burst_Buffer}}
|-
|{{Up | HPSS|HPSS}}
|{{Up3 | Login Nodes|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|{{Up3 | External Network|https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Logging_in}}
|{{Up | Globus |Globus}}
|-
|{{Down | Balam|Balam}}
|{{Up3 | Cvmfs|https://docs.alliancecan.ca/wiki/Standard_software_environments}}
|}

'''Sat Nov 29, 2025, 00:40 am:''' There has been a problem with the water chiller. Some systems are offline.

'''Wed Nov 5, 2025, 12:55 pm:''' Balam is back online.

'''Wed Nov 5, 2025, 10:00 am:''' Open OnDemand is back online.

'''Tue Nov 4, 2025, 11:00 pm:''' Most of the work is done, data movers, Globus, and HPSS are back online. Remaining services will be worked on tomorrow.

'''Tue Nov 4, 2025, 8:30 am:''' Scheduled network maintenance. Trillium cluster is *not* affected.

'''Tue Oct 21, 2025, 17:30 am:''' Balam maintenance finished.

'''Tue Oct 21, 2025, 7:00 am:''' Balam maintenance day.

'''Thu Oct 15, 2025, 3:55 pm:''' Trillium inbound connections through trillium.alliancecan.ca or trillium.scinet.utoronto.ca are working again.

'''Thu Oct 15, 2025, 3:05 pm:''' Trillium is experiencing external network issues for both incoming traffic. Please try: ssh USERNAME@tri-login01.scinet.utoronto.ca in the meantime.

'''Thu Oct 06, 2025, 8:00 pm:''' HPSS is fully functional. You may submit archive jobs from trillium login nodes, datamovers and robots.

'''Thu Oct 03, 2025, 6:30 pm:''' HPSS is back online, and already accessible via alliancecan#hpss Globus endpoint. Directory tree now follows the other Alliance clusters. We're still working on job submission via Slurm

'''Thu Oct 01, 2025, 0:00 am:''' Niagara compute nodes are now unavailable for regular users. The login nodes will remain available for a while to allow a few last data transfers, although transfers from the Niagara file systems to Trillium are best done on nia-dm1.scinet.utoronto.ca.

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

HPSS

2025-10-18T16:48:29Z

Pinto: /* HTAR */

{|align=right
|align=center|'''Topology Overview'''
|-
|[[Image:HPSS-overview.jpg|right|x600px]]
|-
|}

== High Performance Storage System ==

The High Performance Storage System ([https://www.hpss-collaboration.org/index.shtml HPSS] [https://en.wikipedia.org/wiki/High_Performance_Storage_System wikipedia]) is a tape-backed hierarchical storage system that provides a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data can be returned to the active shared parallel filesystem on Trillium ("VAST") when it is needed.

In the context of the {{Alliance}} Resource Allocations, it is called 'nearline'.

Since this system is intended for large data storage, it is accessible only to groups who have been awarded storage space at SciNet beyond 5TB in the yearly RAC resource allocation round. However, upon request, any user may be awarded access to HPSS, up to 2TB per group, so that you may get familiar with the system (just email support@scinet.utoronto.ca)

Access and transfer of data into and out of HPSS is done under the control of the user, whose interaction is expected to be scripted and submitted as a batch job, using one or more of the following utilities:
* [https://www.racf.bnl.gov/Facility/HPSS/Documentation/HSI/doc_intro.html HSI] is a client with an ftp-like functionality which can be used to archive and retrieve large files. It is also useful for browsing the contents of HPSS.
* [https://www.sdcc.bnl.gov/sites/default/files/2021-09/htar.txt HTAR] is a utility that creates tar formatted archives directly into HPSS. It also creates a separate index file (.idx) that can be accessed and browsed quickly.
* [https://support.scinet.utoronto.ca/wiki/index.php/ISH ISH] is a TUI utility that can perform an inventory of the files and directories in your tarballs.

We're currently running HPSS v 11.2.

== Why should I use and trust HPSS? ==
* HPSS is a 25 year-old collaboration between IBM and the DoE labs in the US, and is used by about 45 facilities in the [http://www.top500.org “Top 500”] HPC list (plus some black-sites).
* Over 2.5 ExaBytes of combined storage world-wide.
* The top 3 sites in the World report (fall 2017) having 360PB, 220PB and 125PB in production (ECMWF, UKMO and BNL)
* Environment Canada also adopted HPSS in 2017 to store Nav Canada data as well as to serve as their own archive. Currently has 2 X 100PB capacity installed.
* The SciNet HPSS system has been providing nearline capacity for important research data in Canada since early 2011, already at 10PB levels in 2018
* Very reliable, data redundancy and data insurance built-in (dual copies of everything are kept on tapes at SciNet)
* Data on cache and tapes can be geo-distributed for further resilience and HA.
* Highly scalable; current performance at SciNet - after the hardware and software upgrades in 2019 - Ingest and Recalls ~150 TB/day aggregated.
* HSI/HTAR clients also very reliable and used on several HPSS sites. ISH was written at SciNet.
* [[Media:HPSS_rationale_SNUG.pdf|HPSS fits well with the Storage Capacity Expansion Plan at SciNet]] (pdf presentation)

== Guidelines ==
* A large portion of the storage capacity of HPSS is provided on tape -- a media that is not suited for storing small files. Files smaller than ~200MB should be grouped into tarballs with '''tar''' or '''htar'''.
* If you intent to use '''HSI''' or '''Globus''' to ingest material, you should favor tarballs over expanded directory trees. Your average file size on archive should be over 1GB/file.
* Optimal performance for aggregated transfers and allocation on tapes is obtained with [[Why not tarballs too large |tarballs of size 1TB or less]], whether ingested by htar or hsi ([[Why not tarballs too large |for good reasons]])
* We strongly urge that you use the sample scripts we are providing as the basis for your job submissions.
* Make sure to check the exit codes of each of the steps in your scripts, and check any returned logs for errors after any data transfer or tarball creation process.

== '''#### New to the HPSS/nearline/archive System? ####''' ==

First, nearline, archive, and HPSS all mean the same thing at SciNet. HPSS is the true name of this storage system.

Second, HPSS is not a regular file system, and as such is not mounted on the regular login, compute, and datamover nodes. Interaction with the HPSS system is therefore different than using simple copy, move, and tar commands.

Before trying to use the system, you need to confirm you are part of a group that already has a HPSS RAC allocation, otherwise, email SciNet support and request an HPSS account (or else you will get "Error - authentication/initialization failed" and 71 exit codes).

THIS set of instructions on the wiki is the best and most compressed "manual" we have on how to use HPSS. It may seem a bit overwhelming at first, because of all the job script templates we make available below (they are here so you don't have to think
too much, just copy and paste), but if you approach the index at the top as a "case switch" mechanism for what you intend to do, everything falls in place.

== The 3 ways to access HPSS ==
* Submitting jobs to the [[HPSS#Access_Through_the_Queue_System | Trillium archive partitions]] (archiveshort or archivelong), and using HTAR or HSI on the archive02 node. These tools are optimized for speed and scale, to efficiently deal with small files, and are very resilient to timeouts due to slow tape access. That is our recommendation for large transfers.
* Using Globus, which can be a very good trade off between the efficiency of HSI/HTAR and the convenience of a WebGUI. You may transfer data between 2 endpoints. For instance:
alliancecan#trillium
alliancecan#hpss
* Submitting an interactive job to the [[HPSS#Access_Through_the_Queue_System | VFS partition]] (vfsshort), and using standard linux tools (cp, rsync, ls, cd, ...) to access the VFS based /archive mount point on the vfs02 node. We don't recommend this way, unless you are transferring only a handful of files, or just navigating the naming-space (directory tree similar to the other Trillium file systems).
/archive

Try this sequence:

1) [https://docs.scinet.utoronto.ca/index.php/HPSS#Access_Through_an_Interactive_HSI_session take a look around HPSS using an interactive HSI session]

(most linux shell commands have an equivalent in HPSS)

2) [https://docs.scinet.utoronto.ca/index.php/HPSS#Sample_tarball_create archive a small test directory using HTAR]

2a) use step 1) to see what happened

3) [https://docs.scinet.utoronto.ca/index.php/HPSS#Sample_data_offload archive a file using hsi]

3a) use step 1) to see what happened

4) [https://docs.scinet.utoronto.ca/index.php/HPSS#Sample_transferring_directories archive a small test directory using HSI]

4a) use step 1) to see what happened

5) now try the other cases and so on. In a couple of hours you'll be in pretty good shape.

== Access Through the Queue System ==
All access to the archive system is done through the [https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Submitting_jobs_to_the_scheduler Trillium queue system].

* Job submissions should be done to the 'archivelong', 'archiveshort' or 'vfsshort' partitions.
* Short jobs are limited to 1H walltime by default. Long jobs (> 1H) are limited to 72H walltime.
* Users are limited to only 2 long jobs and 2 short jobs at the same time, and 10 jobs total on the each queue.
* There can only be 5 long jobs running at any given time overall. Remaining submissions will be placed on hold for the time being. So far we have not seen a need for overall limit on short jobs.
* Jobs to the 'archivelong' and 'archiveshort' partitions must use [[HPSS#HTAR | htar]] or [[HPSS#HSI | hsi]]. These tools are optimized for speed and scale, to efficiently deal with small files, and are very resilient to timeouts due to slow tape access.
* The 'vfsshort' partition can only be used for interactive jobs, so that you can navigate the VFS based /archive mount point (Virtual File System), and/or make small file transfers to/from HPSS using standard linux tools (ls, cd, cp, rsync, etc). This access is always limited to 1 hour.

You can submit your job like this:
<pre>
sbatch jobscript.sh (to 'archivelong' or 'archiveshort')
</pre>

And you can get an interactive session like this:
<pre>
salloc -p archiveshort
OR
salloc -p vfsshort
</pre>

[https://docs.alliancecan.ca/wiki/Trillium_Quickstart#Submitting_jobs_to_the_scheduler You may learn SLURM in more details here.]

The status of pending jobs can be monitored with squeue specifying the archive partition:
<pre>
squeue -p archiveshort
OR
squeue -p archivelong
OR
squeue -p vfsshort
</pre>

== Access Through an Interactive HSI session ==
* You may want to acquire an interactive shell, start an HSI session and navigate the archive naming-space. Keep in mind, you're restricted to 1H.

<pre>
pinto@tri-login02:~$ salloc -p archiveshort -t 1:00:00
salloc: Granted job allocation 50918
salloc: Waiting for resource configuration
salloc: Nodes hpss-archive02-ib are ready for job
hpss-archive02-ib:~$

hpss-archive02-ib:~$ hsi (DON'T FORGET TO START HSI)
******************************************************************
* Welcome to HPSS@SciNet - High Perfomance Storage System *
* *
* INFO: THIS IS THE NEW 7.5.1 HPSS SYSTEM! *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
* CHECK THE INTEGRITY OF YOUR TARBALLS *
******************************************************************
[HSI]/archive/scinet/pinto-> ls

[HSI]/archive/scinet/pinto-> cd <some directory>
</pre>

NOTE: the VFS based /archive mount point is not visible on the archive02 node with the standard linux prompt. You must use HSI (or get an interactive session on the vfsshort partition).

=== Scripted File Transfers ===
File transfers in and out of the HPSS should be scripted into jobs and submitted to the ''archivelong'' partition or the ''archiveshort'' partition. See generic example below:

<source lang="bash">
#!/bin/bash -l
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J htar_create_tarball_in_hpss
#SBATCH --mail-type=ALL

echo "Creating a htar of finished-job1/ directory tree into HPSS"
trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

DEST=$ARCHIVE/finished-job1.tar

# htar WILL overwrite an existing file with the same name so check beforehand.

hsi ls $DEST &> /dev/null
status=$?

if [ $status == 0 ]; then
echo 'File $DEST already exists. Nothing has been done'
exit 1
fi

cd $SCRATCH/workarea/
htar -Humask=0137 -cpf $ARCHIVE/finished-job1.tar finished-job1/
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>
'''Note:''' Always trap the execution of your jobs for abnormal terminations, and be sure to return the exit code

=== Job Dependencies ===

Typically data will be recalled to /scratch when it is needed for analysis. Job dependencies can be constructed so that analysis jobs wait in the queue for data recalls before starting. The qsub flag is
<pre>
--dependency=<type:JOBID>
</pre>
where JOBID is the job number of the archive recalling job that must finish successfully before the analysis job can start.

Here is a short cut for generating the dependency (lookup [https://docs.scinet.utoronto.ca/index.php/HPSS#Sample_data_recall data-recall.sh samples]):
<pre>
hpss-archive02-ib:~$ sbatch -d afterok:$(sbatch --parsable data-recall.sh) job-to-work-on-recalled-data.sh
</pre>

== HTAR ==
''' Please aggregate small files (<~200MB) into tarballs or htar files. '''

''' [[Why not tarballs too large |Keep your tarballs to size 1TB or less]], whether ingested by htar or hsi ([[Why not tarballs too large | WHY?]])'''

HTAR is a utility that is used for aggregating a set of files and directories, by using a sophisticated multithreaded buffering scheme to write files directly from VAST into HPSS, creating an archive file that conforms to the POSIX TAR specification, thereby achieving a high rate of performance. HTAR does not do gzip compression, however it already has a built-in checksum algorithm.

'''Caution'''
* Files larger than 68 GB cannot be stored in an HTAR archive (unfortunately old 32-bit limitation). If you attempt to start a transfer with any files larger than 68GB the whole HTAR session will fail, and you'll get a notification listing all those files, so that you can transfer them with HSI.
* Files with pathnames too long will be skipped (greater than 100 characters), so as to conform with TAR protocol [[(POSIX 1003.1 USTAR)]] -- Note that the HTAR will erroneously indicate success, however will produce exit code 70. For now, you can check for this type of error by "grep WARNING my.output" after the job has completed.
* With the consolidation and centralization of all {{Alliance}} accounts under CCDB, the new uid/gid numbers are now in the 3000000-6000000 range, in breach of same above TAR posix protocol, so htar will issue a benign notification similar to this: ''ERROR: [Uint32_tToOctal]Octal field [thb_uid] overflow - width=8 value=3000195''. Not to worry if the jot still gets a successful exit signal from the execution script.
* Unlike with cput/cget in HSI, "prompt before overwrite", this is not the default with (h)tar. Be careful not to unintentionally overwrite a previous htar destination file in HPSS. There could be a similar situation when extracting material back into VAST and overwriting the originals. Be sure to double-check the logic in your scripts.
* Check the HTAR exit code and log file before removing any files from the VAST active filesystems.

=== HTAR Usage ===
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, and preserve mask attributes (-p), enter:
<pre>
htar -cpf files.tar file1 file2
OR
htar -cpf $ARCHIVE/files.tar file1 file2
</pre>

* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:
<pre>
htar -cpf subdirA.tar subdirA/
</pre>

* To extract all files from the archive file called ''proj1.tar'' in HPSS into the ''project1/src'' directory in VAST, and use the time of extraction as the modification time, enter:
<pre>
cd project1/src
htar -xpmf proj1.tar
</pre>

* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):
<pre>
htar -vtf out.tar
</pre>

* To ensure that both the htar and the .idx files have read permissions to other members in your group use the umask option
<pre>
htar -Humask=0137 ....
</pre>

For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online

==== Sample tarball create ====
<source lang="bash">
#!/bin/bash -l
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J htar_create_tarball_in_hpss
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

DEST=$ARCHIVE/finished-job1.tar

# htar WILL overwrite an existing file with the same name so check beforehand.

hsi ls $DEST &> /dev/null
status=$?

if [ $status == 0 ]; then
echo 'File $DEST already exists. Nothing has been done'
exit 1
fi

cd $SCRATCH/workarea/
htar -Humask=0137 -cpf $DEST finished-job1/
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

'''Note:''' If you attempt to start a transfer with any files larger than 68GB the whole HTAR session will fail, and you'll get a notification listing all those files, so that you can transfer them with HSI.
<pre>
----------------------------------------
INFO: File too large for htar to handle: finished-job1/file1 (86567185745 bytes)
INFO: File too large for htar to handle: finished-job1/file2 (71857244579 bytes)
ERROR: 2 oversize member files found - please correct and retry
ERROR: [FATAL] error(s) generating filename list
HTAR: HTAR FAILED
###WARNING htar returned non-zero exit status
</pre>

==== Sample tarball list ====
<source lang="bash">
#!/bin/bash -l
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J htar_list_tarball_in_hpss
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

DEST=$ARCHIVE/finished-job1.tar

htar -tvf $DEST
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Sample tarball extract ====
<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J htar_extract_tarball_from_hpss
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

cd $SCRATCH/recalled-from-hpss
htar -xpmf $ARCHIVE/finished-job1.tar
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

== HSI ==

HSI may be the primary client with which some users will interact with HPSS. It provides an ftp-like interface for archiving and retrieving tarballs or [https://docs.scinet.utoronto.ca/index.php/HPSS#Sample_transferring_directories directory trees]. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents in HPSS. The most commonly used commands will be:
{|border="1" cellpadding="10" cellspacing="0"
|-
| cput
| Conditionally saves or replaces a HPSSpath file to VASTpath if the VAST version is new or has been updated
cput [options] VASTpath [: HPSSpath]
|-
| cget
| Conditionally retrieves a copy of a file from HPSS to VAST only if a VAST version does not already exist.
cget [options] [VASTpath :] HPSSpath
|-
| cd,mkdir,ls,rm,mv
| Operate as one would expect on the contents of HPSS.
|-
| lcd,lls
| ''Local'' commands to VAST
|}

*There are 3 distinctions about HSI that you should keep in mind, and that can generate a bit of confusion when you're first learning how to use it:
** HSI doesn't currently support renaming directories paths during transfers on-the-fly, therefore the syntax for cput/cget may not work as one would expect in some scenarios, requiring some workarounds.
** HSI has an operator ":" which separates the VASTpath and HPSSpath, and must be surrounded by whitespace (one or more space characters)
** The order for referring to files in HSI syntax is different from FTP. In HSI the general format is always the same, VAST first, HPSS second, cput or cget:
<pre>
VASTfile : HPSSfile
</pre>
For example, when using HSI to store the tarball file from VAST into HPSS, then recall it to VAST, the following commands could be used:
<pre>
cput tarball-in-VAST : tarball-in-HPSS
cget tarball-recalled : tarball-in-HPSS
</pre>

unlike with FTP, where the following syntax would be used:
<pre>
put tarball-in-VAST tarball-in-HPSS
get tarball-in-HPSS tarball-recalled
</pre>
* Simple commands can be executed on a single line.
<pre>
hsi "mkdir LargeFilesDir; cd LargeFilesDir; cput tarball-in-VAST : tarball-in-HPSS"
</pre>

* More complex sequences can be performed using an except such as this:
<pre>
hsi <<EOF
mkdir LargeFilesDir
cd LargeFilesDir
cput tarball-in-VAST : tarball-in-HPSS
lcd $SCRATCH/LargeFilesDir2/
cput -Ruph *
end
EOF
</pre>

* The commands below are equivalent, but we recommend that you always use full path, and organize the contents of HPSS, where the default HSI directory placement is $ARCHIVE:
<pre>
hsi cput tarball
hsi cput tarball : tarball
hsi cput $SCRATCH/tarball : $ARCHIVE/tarball
</pre>

* There are no known issues renaming files on-the-fly:
<pre>
hsi cput $SCRATCH/tarball1 : $ARCHIVE/tarball2
hsi cget $SCRATCH/tarball3 : $ARCHIVE/tarball2
</pre>

* However the syntax forms such as the ones below will fail, since they rename the directory paths.
<pre>
hsi cput -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir (FAILS)
OR
hsi cget -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir2 (FAILS)
OR
hsi cput -Ruph $SCRATCH/LargeFilesDir/* : $ARCHIVE/LargeFilesDir2 (FAILS)
OR
hsi cget -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir (FAILS)
</pre>

One workaround is the following 2-steps process, where you do a "lcd " in VAST first, and recursively transfer the whole directory (-R), keeping the same name. You may use '-u' option to resume a previously disrupted session, and the '-p' to preserve timestamp, and '-h' to keep the links.
<pre>
hsi <<EOF
lcd $SCRATCH
cget -Ruph LargeFilesDir
end
EOF
</pre>

Another workaround is do a "lcd" into the VASTpath first and a "cd" in the HPSSpath, but transfer the files individually with the '*' wild character. This option lets you change the directory name:
<pre>
hsi <<EOF
lcd $SCRATCH/LargeFilesDir
mkdir $ARCHIVE/LargeFilesDir2
cd $ARCHIVE/LargeFilesDir2
cput -Ruph *
end
EOF
</pre>

=== Documentation ===
Complete documentation on HSI is available from the Gleicher Enterprises links below. You may peruse those links and come with alternative syntax forms. You may even be already familiar with HPSS/HSI from other HPC facilities, that may or not have procedures similar to ours. HSI doesn't always work as expected when you go outside of our recommended syntax, so '''we strongly urge that you use the sample scripts we are providing as the basis''' for your job submissions
* [http://www.mgleicher.us/index.html/hsi/hsi_reference_manual_2/introduction.html HSI Introduction] (original site inactive)
* [http://www.mgleicher.us/index.html/hsi/hsi_man_page.html man hsi] (original site inactive)
* [https://docs.scinet.utoronto.ca/index.php/HSI_help hsi help]
* [http://www.mgleicher.us/index.html/hsi/hsi-exit-codes.html exit codes] (original site inactive)
'''Note:''' HSI returns the highest-numbered exit code, in case of multiple operations in the same hsi session. You may use '/scinet/hpss/bin/exit2msg $status' to translate those codes into intelligible messages

=== Typical Usage Scripts===
The most common interactions will be ''putting'' data into HPSS, examining the contents (ls,ish), and ''getting'' data back onto VAST for inspection or analysis.

==== Sample data offload ====
<source lang="bash">
#!/bin/bash
# This script is named: data-offload.sh
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J offload
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# individual tarballs already exist

/usr/local/bin/hsi -v <<EOF1
mkdir put-away
cd put-away
cput $SCRATCH/workarea/finished-job1.tar.gz : finished-job1.tar.gz
end
EOF1
status=$?
if [ ! $status == 0 ];then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

/usr/local/bin/hsi -v <<EOF2
mkdir put-away
cd put-away
cput $SCRATCH/workarea/finished-job2.tar.gz : finished-job2.tar.gz
end
EOF2
status=$?
if [ ! $status == 0 ];then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

trap - TERM INT
</source>

'''Note:''' as in the above example, we recommend that you capture the (highest-numbered) exit code for each hsi session independently. And remember, you may improve your exit code verbosity by adding the excerpt below to your scripts:
<source lang="bash">
if [ ! $status == 0 ];then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Sample data list ====
A very trivial way to list the contents of HPSS would be to just submit the HSI 'ls' command.
<source lang="bash">
#!/bin/bash
# This script is named: data-list.sh
#SBATCH -t 1:00:00
#SBATCH -p archiveshort
#SBATCH -N 1
#SBATCH -J hpss_ls
#SBATCH --mail-type=ALL

/usr/local/bin/hsi -v <<EOF
cd put-away
ls -R
end
EOF
</source>
''Warning: if you have a lot of files, the ls command will take a long time to complete. For instance, about 400,000 files can be listed in about an hour. Adjust the walltime accordingly, and be on the safe side.''

However, we provide a much more useful and convenient way to explore the contents of HPSS with the inventory shell [[ISH]]. This example creates an index of all the files in a user's portion of the namespace. The list is placed in the directory /home/$(whoami)/.ish_register that can be inspected from the login nodes.
<source lang="bash">
#!/bin/bash
# This script is named: data-list.sh
#SBATCH -t 1:00:00
#SBATCH -p archiveshort
#SBATCH -N 1
#SBATCH -J hpss_index
#SBATCH --mail-type=ALL

INDEX_DIR=$HOME/.ish_register
if ! [ -e "$INDEX_DIR" ]; then
mkdir -p $INDEX_DIR
fi

export ISHREGISTER="$INDEX_DIR"
/scinet/hpss/bin/ish hindex
</source>
''Note: the above warning on collecting the listing for many files applies here too.''

This index can be browsed or searched with ISH on the development nodes.
<source lang="bash">
hpss-archive02-ib:~$ /scinet/hpss/bin/ish ~/.ish_register/hpss.igz
[ish]hpss.igz> help
</source>

ISH is a powerful tool that is also useful for creating and browsing indices of tar and htar archives, so please look at the [[ISH|documentation]] or built in help.

==== Sample data recall ====
<source lang="bash">
#!/bin/bash
# This script is named: data-recall.sh
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J recall_files
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
mkdir -p $SCRATCH/recalled-from-hpss

# individual tarballs previously organized in HPSS inside the put-away-on-2010/ folder
hsi -v << EOF
cget $SCRATCH/recalled-from-hpss/Jan-2010-jobs.tar.gz : $ARCHIVE/put-away-on-2010/Jan-2010-jobs.tar.gz
cget $SCRATCH/recalled-from-hpss/Feb-2010-jobs.tar.gz : $ARCHIVE/put-away-on-2010/Feb-2010-jobs.tar.gz
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

We should emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do optimization, as in the following example:
<source lang="bash">
#!/bin/bash
# This script is named: data-recall.sh
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J recall_files_optimized
#SBATCH --mail-type=AL

trap "echo 'Job script not completed';exit 129" TERM INT
mkdir -p $SCRATCH/recalled-from-hpss

# individual tarballs previously organized in HPSS inside the put-away-on-2010/ folder
hsi -v << EOF
lcd $SCRATCH/recalled-from-hpss/
cd $ARCHIVE/put-away-on-2010/
cget Jan-2010-jobs.tar.gz Feb-2010-jobs.tar.gz
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

=== Sample transferring directories ===
Remember, it's not possible to rename directories or paths on-the-fly:
<pre>
hsi cget -Ruph $SCRATCH/LargeFiles-recalled : $ARCHIVE/LargeFiles (FAILS)
</pre>

One workaround is transfer the whole directory (and sub-directories) recursively:
<source lang="bash">
#!/bin/bash
# This script is named: data-recall.sh
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J recall_directories
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
mkdir -p $SCRATCH/recalled

hsi -v << EOF
lcd $SCRATCH/recalled
cd $ARCHIVE/
cget -Ruph LargeFiles
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

Another workaround is to transfer files and subdirectories individually with the "*" wild character:
<source lang="bash">
#!/bin/bash
# This script is named: data-recall.sh
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J recall_directories
#SBATCH --mail-type=AL

trap "echo 'Job script not completed';exit 129" TERM INT
mkdir -p $SCRATCH/LargeFiles-recalled

hsi -v << EOF
lcd $SCRATCH/LargeFiles-recalled
cd $ARCHIVE/LargeFiles
cget -Ruph *
end
EOF
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help''']

== [[ISH|ISH]] ==
=== [[ISH|Documentation and Usage]] ===

== File and directory management ==
=== Moving/renaming ===
* you may use 'mv' or 'cp' in the same way as the linux version.
<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J deletion_script
#SBATCH --mail-type=ALL

echo "HPSS file and directory management"
trap "echo 'Job script not completed';exit 129" TERM INT

/usr/local/bin/hsi -v <<EOF1
mkdir $ARCHIVE/2011
mv $ARCHIVE/oldjobs $ARCHIVE/2011
cp -r $ARCHIVE/almostfinished/*done $ARCHIVE/2011
end
EOF1
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

=== Deletions ===
==== Recommendations ====
* Be careful with the use of 'cd' commands to non-existing directories before the 'rm' command. Results may be unpredictable
* Avoid the use of the stand alone wild character '''*'''. If necessary, whenever possible have it bound to common patterns, such as '*.tmp', so to limit unintentional mis-happens
* Avoid using relative paths, even the env variable $ARCHIVE. Better to explicitly expand the full paths in your scripts
* Avoid using recursive/looped deletion instructions on $SCRATCH contents from the archive job scripts. Even on $ARCHIVE contents, it may be better to do it as an independent job submission, after you have verified that the original ingestion into HPSS finished without any issues.

==== Typical example ====
<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J deletion_script
#SBATCH --mail-type=ALL

echo "Deletion of an outdated directory tree into HPSS"

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

/usr/local/bin/hsi -v <<EOF1
rm /archive/scinet/pinto/*.tmp
rm -R /archive/scinet/pinto/obsolete
end
EOF1
status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Deleting with an interactive HSI session ====
* You may feel more comfortable acquiring an interactive shell, starting an HSI session and proceeding with your deletions that way. Keep in mind, you're restricted to 1H.

* After using the ''sbatch -p archiveshort'' command you'll get a standard shell prompt on an archive execution node (hpss-archive02), as you would on any compute node. However you will need to run '''HSI''' or '''HTAR''' to access resources on HPSS.

* HSI will give you a prompt very similar to a standard shell, where your can navigate around using commands such 'ls', 'cd', 'pwd', etc ... NOTE: not every bash command has an equivalent on HSI - for instance, you can not 'vi' or 'cat'.

<pre>
pinto@nia-login07:~$ salloc -p archiveshort -t 1:00:00
salloc: Granted job allocation 50359
salloc: Waiting for resource configuration
salloc: Nodes hpss-archive02-ib are ready for job

hpss-archive02-ib:~$ hsi
******************************************************************
* Welcome to HPSS@SciNet - High Perfomance Storage System *
* *
* INFO: THIS IS THE NEW 7.5.1 HPSS SYSTEM! *
* *
* Contact Information: support@scinet.utoronto.ca *
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *
* CHECK THE INTEGRITY OF YOUR TARBALLS *
******************************************************************

[HSI]/archive/scinet/pinto-> rm -R junk

</pre>

== HPSS for the 'Watchmaker' ==
=== Efficient alternative to htar ===
<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J tar_create_tarball_in_hpss_with_hsi_by_piping
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

# When using a pipeline like this
set -o pipefail

# to put (cput will fail)
tar -c $SCRATCH/mydir | hsi put - : $ARCHIVE/mydir.tar
status=$?
if [ ! $status == 0 ]; then
echo 'TAR+HSI+piping returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# to immediately generate an index
ish hindex $ARCHIVE/mydir.tar
status=$?
if [ ! $status == 0 ]; then
echo 'ISH returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# to get
#cd $SCRATCH
#hsi cget - : $ARCHIVE/mydir.tar | tar -xv
#status=$?
# if [ ! $status == 0 ]; then
# echo 'TAR+HSI+piping returned non-zero code.'
# /scinet/hpss/bin/exit2msg $status
# exit $status
#else
# echo 'TRANSFER SUCCESSFUL'
#fi

trap - TERM INT

</source>
'''Notes:'''
* Combining commands in this fashion, besides being HPSS-friendly, should not be that noticeably slower than the recursive put with HSI that stores each file one by one. However, reading the files back from tape in this format will be many times faster. It would also overcome the current 68GB limit on the size of stored files that we have with htar.
* To top things off, we recommend indexing with ish (in the same script) immediately after the tarball creation , while it resides in the HPSS cache. It would be as if htar was used.
* To ensure that an error at any stage of the pipeline shows up in the returned status use: ''set -o pipefail'' (The default is to return the status of the last command in the pipeline and this is not what you want.)
* Optimal performance for aggregated transfers and allocation on tapes is obtained with [[Why not tarballs too large |tarballs of size 500GB or less]], whether ingested by htar or hsi ([[Why not tarballs too large | WHY?]]). Be sure to check the contents of the directory tree with 'du' for the total amount of data before sending them to the tar+HSI piping.

=== Multi-threaded gzip'ed compression with pigz ===
We compiled multi-threaded implementation of gzip called pigz (http://zlib.net/pigz/). It's now part of the "extras" module. It can also be used on any compute or devel nodes. This makes the execution of the previous version of the script much quicker than if you were to use 'tar -cfz'. In addition, by piggy-backing ISH to the end of the script, it will know what to do with the just created mydir.tar.gz compressed tarball.

<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J tar_create_compressed_tarball_in_hpss_with_hsi_by_piping
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

# When using a pipeline like this
set -o pipefail

# to put (cput will fail)
tar -c $SCRATCH/mydir | pigz | hsi put - : $ARCHIVE/mydir.tar.gz
status=$?
if [ ! $status == 0 ]; then
echo 'TAR+PIGZ+HSI+piping returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

NOTE: Do not use this type of pipeline to serialize recalls with HSI inside a for loop. This is a very inefficient way for HPSS to handle this type of operation, since it forces the system to fetch the tarballs using your sequence on the loop, or a sequence based on a numeric/alphabetic/chronological order from the tapes. Files are not necessarily written to tapes in the same sequence in which they are ingested. There will be many unnecessary mounts/dismounts involved, and a lot of fast forward and rewinds, depending on where the files are on the tapes, which can be extremely stressful to the hardware. This effect is know a "shoe-shinning".

Instead, provide HSI with a full list of tarballs to be recalled, all at once, and the HSI will sort the list in the most convenient order possible, in the sequence that the files are laid out on the tapes, and fetch all the files from each tape at once, in one go.
Afterwards, once all the tarballs are back onto your $SCRATCH, you may use a loop to unpigz the tarballs

=== Content Verification ===

==== HTAR CRC checksums ====
Specifies that HTAR should generate CRC checksums when creating the archive.

<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J htar_create_tarball_in_hpss_with_checksum_verification
#SBATCH --mail-type=ALL

trap "echo 'Job script not completed';exit 129" TERM INT
# Note that your initial directory in HPSS will be $ARCHIVE

cd $SCRATCH/workarea

# to put
htar -Humask=0137 -cpf $ARCHIVE/finished-job1.tar -Hcrc -Hverify=1 finished-job1/

# to get
#mkdir $SCRATCH/verification
#cd $SCRATCH/verification
#htar -Hcrc -xvpmf $ARCHIVE/finished-job1.tar

status=$?

trap - TERM INT

if [ ! $status == 0 ]; then
echo 'HTAR returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Current HSI version - Checksum built-in ====

MD5 is the standard Hashing Algorithm for the HSI build at SciNet. For hsi ingestions with the '-c on' option you should be able to query the md5 hash with the hsi command 'hashli'. That value is stored as an UDA (User Defined Attribute) for each file (a feature of HPSS starting with 7.4)

[http://www.mgleicher.us/GEL/hsi/hsi/hsi_reference_manual_2/checksum-feature.html More usage details here]

The checksum algorithm is very CPU-intensive. Although the checksum code is compiled with a high level of compiler optimization, transfer rates can be significantly reduced when checksum creation or verification is in effect. The amount of degradation in transfer rates depends on several factors, such as processor speed, network transfer speed, and speed of the local filesystem (VAST).

<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J MD5_checksum_verified_transfer
#SBATCH --mail-type=ALL

thefile=<VASTpath>
storedfile=<HPSSpath>

# Generate checksum on fly (-c on)
hsi -q put -c on $thefile : $storedfile
pid=$!

# Check the exit code of the HSI process
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# verify checksum
hsi lshash $storedfile
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# get the file back with checksum
hsi get -c on $storedfile
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

==== Prior to HSI version 4.0.1.1 ====

This will checksum the contents of the HPSSpath against the original VASTpath after the transfer has finished.
<source lang="bash">
#!/bin/bash
#SBATCH -t 72:00:00
#SBATCH -p archivelong
#SBATCH -N 1
#SBATCH -J checksum_verified_transfer
#SBATCH --mail-type=ALL

thefile=<VASTpath>
storedfile=<HPSSpath>

# Generate checksum on fly using a named pipe so that file is only read from VAST once
mkfifo /tmp/NPIPE
cat $thefile | tee /tmp/NPIPE | hsi -q put - : $storedfile &
pid=$!
md5sum /tmp/NPIPE |tee /tmp/$fname.md5
rm -f /tmp/NPIPE

# Check the exit code of the HSI process
wait $pid
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi

# change filename to stdin in checksum file
sed -i.1 "s+/tmp/NPIPE+-+" /tmp/$fname.md5

# verify checksum
hsi -q get - : $storedfile | md5sum -c /tmp/$fname.md5
status=$?

if [ ! $status == 0 ]; then
echo 'HSI returned non-zero code.'
/scinet/hpss/bin/exit2msg $status
exit $status
else
echo 'TRANSFER SUCCESSFUL'
fi
</source>

=== Splitting tarballs ===

By knowing the size of your files or directories you can decide how to divide and organize them in different archives if necessary, so not to create huge tarballs to start with.

However, if you find it more convenient to bundle a whole directory into a big tarball, you can still divide it into small chunks prior to ingesting them into HPSS, by using the following syntax:

<code>split -b <Size-in-GB> <tar-file-name> <prefix-name></code>

<source lang="console">
$ split -b 500GB results.tar small-chunk
</source>

The option '''b''' is invoked to fix the size of the small chunks; and '''prefix-name''' is the base name for the small chunks. The above command will split '''results.tar''' into 500 GB chunks in current working directory. The small chunks will starts from: small-chunka small-chunkb small-chunkc small-chunkd .... etc. You may then ingest them into HPSS using any of the 3 methods describe above (HSI, VFS, Globus).

To rebuild the original tarball, recall the individual chunks from HPSS and use the <code>cat</code> command as follows:

<source lang="console">
$ cat small-chunk* > your_archive_name.tar
</source>

In case you want to append numbers in place of alphabets, use '''-d''' option in above split command.

== Access to HPSS using Globus ==
* You may now transfer data between SciNet's HPSS and an external source
* Follow the link below
https://globus.alliancecan.ca
: Enter your {{Alliance}} username and password (and use your MFA).
* In the 'File Manager' tab, enter ''''alliancecan#hpss'''' as one of the Endpoints. If you are seeing 'Missing required data_access_consent', authenticate this endpoint by clicking on 'Continue', enter your username and password once more.
* You may read more about {{Alliance}}} Globus Portal here:
https://docs.alliancecan.ca/wiki/Globus

== User provided Content/Suggestions ==

== [[HPSS-by-pomes|Packing up large data sets and putting them on HPSS]] ==
(Pomés group recommendations)

[[Data Management|BACK TO Data Management]]

Main Page

2025-10-07T00:07:38Z

Pinto:

Main Page

2025-10-03T22:37:44Z

Pinto: /* System Status */

Main Page

2025-10-03T22:35:45Z

Pinto: /* System Status */

Main Page

2025-10-03T22:33:08Z

Pinto:

Main Page

2025-10-03T22:26:52Z

Pinto:

Main Page

2025-10-01T13:59:20Z

Pinto:

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial | Niagara|Niagara_Quickstart}}
|{{Up | Teach|Teach}}
|{{Up | Rouge|Rouge}}
|-
|{{Up | OnDemand|Open_OnDemand_Quickstart}}
|{{Up | Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up | File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Down | HPSS|HPSS}}
|{{Up | Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up | External Network|Niagara_Quickstart#Logging_in}}
|{{Partial | Globus |Globus}}
|-
|{{Up | Balam|Balam}}
|{{Up | Cvmfs|Using_modules}}
|}

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance, including alliancecan#hpss Globus endpoint

'''Thu Sep 18, 2025, 11:30 am:''' Open OnDemand is fully functional again.

'''Wed Sep 17, 2025, 6:00 pm:''' Niagara is back up as well (including its Globus endpoint). We are still working on the other systems.

'''Wed Sep 17, 2025, 1:40 pm:''' Trillium is back up (except for its Globus endpoint). We are working on the other systems still.

'''Tue Sep 16, 2025, 5:45 pm:''' Unfortunately, we cannot bring all systems up yet because we are waiting for a spare part for the cooling system that will be brought tomorrow. In the meantime, we have managed to keep the Trillium login nodes up, but not other systems.

'''Tue Sep 16, 2025, from 7:00 am to 5:00 pm (EDT):''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Niagara, Mist, HPSS, Rouge, Teach, as well as hosted equipment). This will also be the time that the Mist cluster gets decommissioned.

'''Fri Sep 12 22:03:17 EDT 2025:''' HPSS software and OS upgrades are finished.

'''Tue Sep 9 17:05:38 EDT 2025:''' Starting tomorrow, Sep/10, and for the following 3 days HPSS will be down for software and OS upgrades. We will strive to finish sooner, at which time we will make the system available to users again.

===Mist/Niagara Decommissioning Schedule===

'''September 4, 2025'''
* Niagara reduced to 863 compute nodes.

'''September 9, 2025'''
* Niagara's Open OnDemand decommissioned.
* Brief data centre connection outage at 9 AM EDT
* Niagara reduced to 647 compute nodes at end of day.

'''September 11, 2025'''
* Trillium Open OnDemand goes live.

'''September 16, 2025'''
* '''Full-day data centre maintenance'''
* Niagara reduced to 431 compute nodes.
* Mist decommissioned.

'''September 24, 2025'''
* Niagara reduced to 215 compute nodes.

'''September 30, 2025'''
* Niagara decommissioned.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2025-10-01T13:58:05Z

Pinto:

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial | Niagara|Niagara_Quickstart}}
|{{Up | Teach|Teach}}
|{{Up | Rouge|Rouge}}
|-
|{{Up | OnDemand|Open_OnDemand_Quickstart}}
|{{Up | Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up | File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Down | HPSS|HPSS}}
|{{Up | Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up | External Network|Niagara_Quickstart#Logging_in}}
|{{Up | Globus |Globus}}
|-
|{{Up | Balam|Balam}}
|{{Up | Cvmfs|Using_modules}}
|}

'''Thu Oct 01, 2025, 9:30 am:''' HPSS is down for scheduled maintenance.

'''Thu Sep 18, 2025, 11:30 am:''' Open OnDemand is fully functional again.

'''Wed Sep 17, 2025, 6:00 pm:''' Niagara is back up as well (including its Globus endpoint). We are still working on the other systems.

'''Wed Sep 17, 2025, 1:40 pm:''' Trillium is back up (except for its Globus endpoint). We are working on the other systems still.

'''Tue Sep 16, 2025, 5:45 pm:''' Unfortunately, we cannot bring all systems up yet because we are waiting for a spare part for the cooling system that will be brought tomorrow. In the meantime, we have managed to keep the Trillium login nodes up, but not other systems.

'''Tue Sep 16, 2025, from 7:00 am to 5:00 pm (EDT):''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Niagara, Mist, HPSS, Rouge, Teach, as well as hosted equipment). This will also be the time that the Mist cluster gets decommissioned.

'''Fri Sep 12 22:03:17 EDT 2025:''' HPSS software and OS upgrades are finished.

'''Tue Sep 9 17:05:38 EDT 2025:''' Starting tomorrow, Sep/10, and for the following 3 days HPSS will be down for software and OS upgrades. We will strive to finish sooner, at which time we will make the system available to users again.

===Mist/Niagara Decommissioning Schedule===

'''September 4, 2025'''
* Niagara reduced to 863 compute nodes.

'''September 9, 2025'''
* Niagara's Open OnDemand decommissioned.
* Brief data centre connection outage at 9 AM EDT
* Niagara reduced to 647 compute nodes at end of day.

'''September 11, 2025'''
* Trillium Open OnDemand goes live.

'''September 16, 2025'''
* '''Full-day data centre maintenance'''
* Niagara reduced to 431 compute nodes.
* Mist decommissioned.

'''September 24, 2025'''
* Niagara reduced to 215 compute nodes.

'''September 30, 2025'''
* Niagara decommissioned.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Data Management

2025-09-24T21:24:14Z

Pinto: /* Using mmputacl/mmgetacl */

Understanding the various file systems, and how to use them properly, is critical to optimizing your workflow and being a good SciNet citizen. This page describes the various Niagara file systems, and how to properly use them.

=Performance=
The file systems on SciNet, with the exception of archive, are [http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System GPFS], a high-performance file system which provides rapid reads and writes to large datasets in parallel from many nodes. As a consequence of this design, however, '''the file system performs quite ''poorly'' at accessing data sets which consist of many, small files.''' For instance, you will find that reading data in from one 16MB file is enormously faster than from 400 40KB files. Such small files are also quite wasteful of space, as the [https://en.wikipedia.org/wiki/Block_(data_storage) blocksize] for the scratch and project filesystems is 16MB. This is something you should keep in mind when planning your input/output strategy for runs on SciNet.

For instance, if you run multi-process jobs, having each process write to a file of its own is not an scalable I/O solution. A directory gets locked by the first process accessing it, so all other processes have to wait for it. Not only has the code just become considerably less parallel, chances are the file system will have a time-out while waiting for your other processes, leading your program to crash mysteriously.
Consider using MPI-IO (part of the MPI-2 standard), which allows files to be opened simultaneously by different processes, or using a dedicated process for I/O to which all other processes send their data, and which subsequently writes this data to a single file.

= Purpose of each file system =

Niagara, Mist and Rouge accesses several different file systems. Note that not all of these file systems are available to all users.

== /home ($HOME) ==
/home is intended primarily for individual user files, common software or small datasets used by others in the same group, provided it does not exceed individual quotas. Otherwise you may consider /scratch or /project. /home is read-only on the compute nodes and has daily backups.

== /scratch ($SCRATCH) ==
/scratch is to be used primarily for temporary or transient files, checkpoint dumps, for all the results of your computations and simulations, or any material that can be easily recreated or reacquired. You may use scratch as well for any intermediate step in your workflow, provided it does not induce too much I/O (Input/Output) or too many small files on this disk-based storage pool, otherwise you should consider burst buffer (/bb). Once you have your final results, those that you want to keep for the long term, you may migrate them to /project or /archive. /scratch is purged on a regular basis and has no backups.

== /project ($PROJECT) ==
/project is available to groups whose PIs have a storage allocation, and is intended for common group software, large static datasets, or any material very costly to be reacquired or re-generated by the group, associated with jobs currently running on niagara or mist. Material on /project is expected to remain relatively immutable over time. '''You should think of $PROJECT as if it was $STATIC.''' Temporary or transient files should be kept on scratch, not project. High data turnover induces stress and unnecessary consumption of tapes on the TSM backup system, long after this material has been deleted, due to backup retention policies and the extra versions kept of the same file. Even renaming top directories is enough to trick the system into assuming a completely new directory tree has been created, and the old one deleted, hence think carefully about your naming convention ahead of time, and stick with it. Users abusing the project file system and using it as scratch will be flagged and contacted. Note that on niagara /project is only available to groups with RAC allocation.

== /bb ($BBUFFER) ==
/bb, the [[Burst_Buffer| burst buffer]], is a very fast, very high performance alternative to /scratch, made of solid-state drives (SSD). You may request this resource if you anticipate a lot of IOPs (Input/Output Operations) or when you notice your job is not performing well running on scratch or project because of I/O (Input/Output) bottlenecks. See [[Burst_Buffer|here]] for more details.

== /archive ($ARCHIVE) ==
/archive is available to groups whose PIs have a storage allocation, and on niagara it is the 'nearline' storage pool. It's used if you want to temporarily offload semi-active material from any of the above file systems. In practice users will offload/recall material as part of their regular workflow, or when they hit their quotas on scratch or project. That material can remain on HPSS for a few months to a few years. Note that on niagara /archive is only available to groups with RAC allocation.

== /dev/shm (RAM) ==
On the Niagara nodes a [[User_Ramdisk | ramdisk]] is available. [[User_Ramdisk | Ramdisk]] is much faster than real disk, and faster than Burst Buffer. Up to 70 percent of the RAM on the node (i.e. 202GB) may be used as a temporary '''local''' file system. This is particularly useful in the early stages of migrating desktop-computing codes to a HPC platform such as Niagara, especially those that use a lot of file I/O (Input/Output). Using a lot of I/O is a bottleneck in large scale computing, especially on parallel file systems (such as the GPFS used on Niagara), since the files are synchronized across the whole network.

== $SLURM_TMPDIR (RAM) ==
For consistency with the general purpose clusters (Cedar, Graham, Beluga and Narval), the environment variable $SLURM_TMPDIR will be set on Niagara compute jobs. Note that this variable will point to RAMdisk, not to local hard drives. The $SLURM_TMPDIR directory will be empty when your jobs starts and its content gets deleted after the job has finished.


Note that Niagara compute nodes have no local disks, so <tt>$SLURM_TMPDIR</tt> lives in memory (ramdisk), in contrast to the general purpose systems of the {{Alliance}}, i.e., Cedar, Graham, Beluga and Narval, where this variable points to a directory on a node-local ssd disk.

= Quotas and purging =

You should familiarize yourself with the [[Data_Management#Purpose_of_each_file_system | various file systems]], what purpose they serve, and how to properly use them. This table summarizes the various file systems.

{| class="wikitable"
! location
!colspan="2"| quota
!align="right"| block size
! expiration time
! backed up
! on login nodes
! on compute nodes
|-
| $HOME
|colspan="2"| 100 GB / 250,000 files per user
|align="right"| 1 MB
|
| yes
| yes
| read-only
|-
|rowspan="6"| $SCRATCH
|colspan="2"| 25 TB / 6,000,000 files per user provided group quota is not reached
|align="right" rowspan="6" | 16 MB
|rowspan="6"| 2 months
|rowspan="6"| no
|rowspan="6"| yes
|rowspan="6"| yes
|-
|align="right"|groups of up to 4 users
|align="right"|50TB for the group
|-
|align="right"|groups of up to 11 users
|align="right"|125TB for the group
|-
|align="right"|groups of up to 28 users
|align="right"|250TB for the group
|-
|align="right"|groups of up to 60 users
|align="right"|400TB for the group
|-
|align="right"|groups with over 60 users
|align="right"|500TB for the group
|-
| $PROJECT
|colspan="2"| by group allocation
|align="right"| 16 MB
|
| yes
| yes
| yes
|-
| $ARCHIVE
|colspan="2"| by group allocation
|align="right"|
|
| dual-copy
| no
| no
|-
| $BBUFFER
|colspan="2"| 10 TB per user
|align="right"| 1 MB
| very short
| no
| yes
| yes
|}

<ul>
<li>[https://docs.scinet.utoronto.ca/images/9/9a/Inode_vs._Space_quota_-_v2x.pdf Inode vs. Space quota (PROJECT and SCRATCH)]</li>
<li>[https://docs.scinet.utoronto.ca/images/0/0e/Scratch-quota.pdf dynamic quota per group (SCRATCH)]</li>
<li>Compute nodes do not have local storage.</li>
<li>Archive space is on [[HPSS|HPSS]], and is not accessible on the Niagara login, compute, or datamover nodes.</li>
<li>Backup means a recent snapshot, not a replica of all data or version that ever was.</li>
<li><code>$BBUFFER</code> stands for the [[Burst Buffer]], a faster parallel storage tier for temporary data.</li></ul>

== How much Disk Space Do I have left? ==

The <tt>'''/scinet/niagara/bin/diskUsage'''</tt> command, available on the login nodes and datamovers, provides information in a number of ways on the home, scratch, project and archive file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
</pre>

Did you know that you can check which of your directories have more than 1000 files with the <tt>'''/scinet/niagara/bin/topUserDirOver1000list'''</tt> command and which have more than 1GB of material with the <tt>'''/scinet/niagara/bin/topUserDirOver1GBlist'''</tt> command?

Note: information on usage and quota is only updated every 3 hours!

== Scratch Disk Purging Policy ==

In order to ensure that there is always sufficient space available for running jobs '''we automatically delete files in /scratch that have not been accessed or modified for more than 2 months by the 15th of each month'''. Note that we recently changed the reference time to be the ''MostRecentOf(atime,ctime)''. This policy is subject to revision depending on its effectiveness. More details about the purging process and how users can check if their files will be deleted follow. If you have files scheduled for deletion you should move them to a more permanent location, such as your departmental server, your /project space or into HPSS (for PIs who have either been allocated storage space by the RAC on project or HPSS).

On the '''first''' of each month, a list of files scheduled for purging is produced, and an email notification is sent to each user on that list. Users also get a shell notification on every login to Niagara. Furthermore, at/or about the '''12th''' of each month a 2nd scan produces a more-current assessment and another email notification is sent. This way users can double check that they have indeed taken care of all the files they needed to relocate before the purging deadline. Those files will be automatically deleted on the '''15th''' of the same month unless they have been accessed or relocated in the interim. If you have files scheduled for deletion then they will be listed in a file in /scratch/t/todelete/current, which has your userid and groupid in the filename. For example, if user xxyz wants to check if they have files scheduled for deletion they can issue the following command on a system which mounts /scratch (e.g. a Niagara login node): '''ls -1 /scratch/t/todelete/current | grep xxyz'''. In the example below, the name of this file indicates that user xxyz is part of group abc, has 9,560 files scheduled for deletion and they take up 1.0TB of space:

<pre>
[xxyz@nia-login03 ~]$ ls -1 /scratch/t/todelete/current |grep xxyz
-rw-r----- 1 xxyz root 1733059 Jan 17 11:46 3110001___xxyz_______abc_________1.00T_____9560files
</pre>

The file itself contains a list of all files scheduled for deletion (in the last column) and can be viewed with standard commands like more/less/cat - e.g.

'''more /scratch/t/todelete/current/3110001___xxyz_______abc_________1.00T_____9560files'''

<pre>
[_inode information__] [uidNumber] [__________atime__________] [__________ctime__________] [size] [_____file_path_____]
659919349 1268424780 0 -u 3199999 -a2019-26-11 08:49:27.745412 -c2019-26-11 08:49:27.739630 -s 234 -- /gpfs/fs0/scratch/...
</pre>

Similarly, you can also verify all other users in your group by using the ls command with grep on your group. For example: '''ls -1 /scratch/t/todelete/current | grep abc'''. That will list all other users in the same group that xxyz is part of, and have files to be purged on the 15th. Members of the same group have access to each other's contents.

If you access/read/move/delete some of the candidates between the 1st and the 11th, there won't be any changes in the assessment until the 12th.

If there was an assessment file up until the 11th, but no longer on the 12th, it's because you don't have anything to be purged anymore.

If you access/read/move/delete some or the candidates after the 12th, then you have to check yourself to confirm your files won't be purged on the 15th (see below)

'''NOTE:''' Preparing these assessments takes several hours. If you change the access/modification time of a file in the interim, that will not be detected until the next cycle. A way for you to get immediate feedback is to use the ''''ls -lu'''' command on the file to verify the ctime and ''''ls -lc'''' for the mtime. If the file atime/ctime has been updated in the meantime, come the purging date on the 15th it will no longer be deleted.

Purging on niagara is final. Purged files can not be recovered.

= Handling large_collections of files =
There are a number of situations in which dealing with a format that encapsulates and compacts very large numbers of small files may be more performant and convenient than handling those small files in expanded format directly in the file system. These alternative file formats also save a lot of stress on the backup system, since tape is not designed to deal with a lot of small files. It also help you not hit your inodes quota on the file systems.

Please visit this page for more details:

https://docs.alliancecan.ca/wiki/Handling_large_collections_of_files

= Backup Policy =
Our backup is based on versions, not on date or age:

* In general we keep the 2 most recent versions of a file, one per day, provided it exists on the file system. Once a file is deleted we expire the oldest version from the backup, and keep the most recent for 60 days. After that grace period then that only version is expired as well.

* We may have 1 or 2 versions of a file on the backup for over 10 years, provided the original has never been deleted from the file system.

* On the other hand we may not have any backup, if the user created the file in the morning and deleted in the afternoon, since the backup system never had a chance to capture the file (it runs once a day around midnight).

* And the user may have generated several versions of a file during the day, however only the most recent version before the backup runs will be captured for that day.

= Moving data =

Data for analysis and final results need to be moved to and from Niagara. There are several ways to accomplish this.

== Using rsync/scp ==

Move amounts less than 10GB through the login nodes.
* Niagara login nodes and datamovers are visible from outside SciNet.
* Use scp or rsync to niagara.scinet.utoronto.ca or niagara.alliancecan.ca (no difference).
* This will time out for amounts larger than about 10GB.

Move amounts larger than 10GB through the datamover nodes.
* From a Niagara login node, ssh to <code>nia-datamover1</code> or <code>nia-datamover2</code>. From there you can transfer to or from Niagara.
* Alternatively, you may also login/scp/rsync directly to the datamovers from the outside:
nia-datamover1.scinet.utoronto.ca
nia-datamover2.scinet.utoronto.ca
* If you do this often, consider using [https://docs.alliancecan.ca/wiki/Globus Globus], a web-based tool for data transfer.

Note that you can only connect 4 times in a 2-minute window to the login nodes or the datamover nodes. So bundle your transfers, i.e., specify multiple files to be copied as arguments to scp or rsync, or copy whole directories, or zip/tar the files up and unzip/untar them on the other end.

If you want to transfer smaller files between other Digital Research Alliance Canada clusters and Niagara use the SSH agent forwarding flag, <code>-A</code> when logging into another cluster. For example, to copy files to Niagara from Cedar use:

<pre>
ssh -A USERNAME@cedar.alliancecan.ca
</pre>
then perform the copy:
<pre>
[USERNAME@cedar5 ~]$ scp file USERNAME@niagara.alliancecan.ca:/scratch/g/group/USERNAME/
</pre>

== Using Globus ==
Please check the comprehensive documentation [https://docs.alliancecan.ca/wiki/Globus here], and [[Globus | here]].

The Niagara endpoint is "computecanada#niagara".

== Moving data to HPSS/Archive/Nearline ==
HPSS is for long-term storage of data.
* [[HPSS]] is a tape-based storage solution, and is SciNet's nearline a.k.a. archive facility.
* Storage space on HPSS is allocated through the annual [https://alliancecan.ca/en/services/advanced-research-computing/accessing-resources/resource-allocation-competition Resource Allocation Competition (RAC) administered by the Digital Research Alliance of Canada].

=File/Ownership Management (ACL)=
* By default, at SciNet, users within the same group already have read permission to each other's files (not write)
* You may use access control list ('''ACL''') to allow your supervisor (or another user within your group) to manage files for you (i.e., create, move, rename, delete), while still retaining your access and permission as the original owner of the files/directories. You may also let users in other groups or whole other groups access (read, execute) your files using this same mechanism.


==Using mmputacl/mmgetacl==
* You may use gpfs' native '''mmputacl''' and '''mmgetacl''' commands. The advantages are that you can set "control" permission and that [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.doc%2Fgpfs31%2Fbl1adm1160.html POSIX or NFS v4 style ACL] are supported. You will need first to create a /tmp/supervisor.acl template with the following contents, as needed:
<pre>
user::rwxc
group::r-xc
other::----
mask::rwxc
user:[owner]:rwxc
user:[supervisor]:rwxc #read and WRITE permissions to supervisor (may not be necessary)
group:[othegroup]:r-xc #read ONLY permissions to members of other groups (recommended)
</pre>

Then issue the following 2 commands:
<pre>
1) $ mmputacl -i /tmp/supervisor.acl /project/g/group/[owner]
2) $ mmputacl -d -i /tmp/supervisor.acl /project/g/group/[owner]
(every *new* file/directory inside [owner] will inherit [supervisor] ownership by default as well as
[owner] ownership, ie, ownership of both by default, for files/directories created by [supervisor])

$ mmgetacl /project/g/group/[owner]
(to determine the current ACL attributes)

$ mmdelacl -d /project/g/group/[owner]
(to remove any previously set ACL)

$ mmeditacl /project/g/group/[owner]
(to create or change a GPFS access control list)
(for this command to work set the EDITOR environment variable: export EDITOR=/usr/bin/vi)
</pre>

If you want to apply ACL to a folder deep in the tree, as in /project/g/group/owner/dir1/subdir2/subdir3, you will need to also apply ACL to every individual path above the subdir3 level, as in:
<pre>
$ mmputacl -i /tmp/supervisor.acl /project/g/group/owner
$ mmputacl -i /tmp/supervisor.acl /project/g/group/owner/dir1
$ mmputacl -i /tmp/supervisor.acl /project/g/group/owner/dir1/subdir2
$ mmputacl -i /tmp/supervisor.acl /project/g/group/owner/dir1/subdir2/subdir3

$ mmputacl -d -i /tmp/supervisor.acl /project/g/group/owner/dir1/subdir2/subdir3
</pre>

In addition, you'll need to ask your PI to apply ACL to the group level:
<pre>
$ mmputacl -i /tmp/supervisor.acl /project/g/group
</pre>
NOTES:
* There is no option to recursively add or remove ACL attributes using a gpfs built-in command to existing files. You'll need to use the -i option as above for each file or directory individually. [[Recursive_ACL_script | Here is a sample bash script you may use for that purpose]]

* mmputacl will not overwrite the original linux group permissions for a directory when copied to another directory already with ACLs, hence the "#effective:r-x" note you may see from time to time with mmgetacf. If you want to give rwx permissions to everyone in your group you should simply rely on the plain unix 'chmod g+rwx' command. You may do that before or after copying the original material to another folder with the ACLs.

* The only latitude you have is with the "w". You may or not want to let the collaborator/supervisor write to your folder. As for "r-xc" you don't have the option, this combination must always be applied.

* In the case of PROJECT, your group's supervisor will need to set proper ACL to the /project/G/GROUP level in order to let users from other groups access your files.

* ACL's won't let you give away permissions to files or directories that do not belong to you.

* We highly recommend that you never give write permission to other users on the top level of your home directory (/home/G/GROUP/[owner]), since that would seriously compromise your privacy, in addition to disable ssh key authentication, among other things. If necessary, make specific sub-directories under your home directory so that other users can manipulate/access files from those.

* Just a reminder: setfacl/getfacl only works on cedar/graham/beluga, since they have lustre. On niagara you have to use the mm* command just for GPFS: mmputacl, mmgetacl, mmdelacl, mmeditacl

For more information on using [https://www.ibm.com/support/knowledgecenter/SSFKCN_4.1.0/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1adm_mmputacl.htm <tt>mmputacl</tt>] or [https://www.ibm.com/support/knowledgecenter/SSFKCN_4.1.0/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1adm_mmgetacl.htm <tt>mmgetacl</tt>] see their man pages.

==Recursive ACL script ==
You may use/adapt '''[[Recursive_ACL_script| this sample bash script]]''' to recursively add or remove ACL attributes using gpfs built-in commands

Courtesy of Agata Disks (http://csngwinfo.in2p3.fr/mediawiki/index.php/GPFS_ACL)

Main Page

2025-09-22T18:59:35Z

Pinto:

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial | Niagara|Niagara_Quickstart}}
|{{Down | Teach|Teach}}
|{{Up | Rouge|Rouge}}
|-
|{{Up | OnDemand|Open_OnDemand_Quickstart}}
|{{Up | Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up | File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up | External Network|Niagara_Quickstart#Logging_in}}
|{{Up | Globus |Globus}}
|-
|{{Up | Balam|Balam}}
|{{Up | Cvmfs|Using_modules}}
|}

'''Thu Sep 18, 2025, 11:30 am:''' Open OnDemand is fully functional again.

'''Wed Sep 17, 2025, 6:00 pm:''' Niagara is back up as well (including its Globus endpoint). We are still working on the other systems.

'''Wed Sep 17, 2025, 1:40 pm:''' Trillium is back up (except for its Globus endpoint). We are working on the other systems still.

'''Tue Sep 16, 2025, 5:45 pm:''' Unfortunately, we cannot bring all systems up yet because we are waiting for a spare part for the cooling system that will be brought tomorrow. In the meantime, we have managed to keep the Trillium login nodes up, but not other systems.

'''Tue Sep 16, 2025, from 7:00 am to 5:00 pm (EDT):''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Niagara, Mist, HPSS, Rouge, Teach, as well as hosted equipment). This will also be the time that the Mist cluster gets decommissioned.

'''Fri Sep 12 22:03:17 EDT 2025:''' HPSS software and OS upgrades are finished.

'''Tue Sep 9 17:05:38 EDT 2025:''' Starting tomorrow, Sep/10, and for the following 3 days HPSS will be down for software and OS upgrades. We will strive to finish sooner, at which time we will make the system available to users again.

===Mist/Niagara Decommissioning Schedule===

'''September 4, 2025'''
* Niagara reduced to 863 compute nodes.

'''September 9, 2025'''
* Niagara's Open OnDemand decommissioned.
* Brief data centre connection outage at 9 AM EDT
* Niagara reduced to 647 compute nodes at end of day.

'''September 11, 2025'''
* Trillium Open OnDemand goes live.

'''September 16, 2025'''
* '''Full-day data centre maintenance'''
* Niagara reduced to 431 compute nodes.
* Mist decommissioned.

'''September 24, 2025'''
* Niagara reduced to 215 compute nodes.

'''September 30, 2025'''
* Niagara decommissioned.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2025-09-18T01:03:16Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Up3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Partial | Niagara|Niagara_Quickstart}}
|{{Down | Teach|Teach}}
|{{Down | Rouge|Rouge}}
|-
|{{Down | OnDemand|Open_OnDemand_Quickstart}}
|{{Up | Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up | File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Up| HPSS|HPSS}}
|{{Up | Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up| External Network|Niagara_Quickstart#Logging_in}}
|{{Partial| Globus |Globus}}
|-
|{{Up | Balam|Balam}}
|{{Up | Cvmfs|Using_modules}}
|}

'''Tue Sep 17, 2025, 6:00 pm:''' Niagara is back up as well (including its Globus endpoint). We are still working on the other systems.

'''Tue Sep 17, 2025, 1:40 pm:''' Trillium is back up (except for its Globus endpoint). We are working on the other systems still.

'''Tue Sep 16, 2025, 5:45 pm:''' Unfortunately, we cannot bring all systems up yet because we are waiting for a spare part for the cooling system that will be brought tomorrow. In the meantime, we have managed to keep the Trillium login nodes up, but not other systems.

'''Tue Sep 16, 2025, from 7:00 am to 5:00 pm (EDT):''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Niagara, Mist, HPSS, Rouge, Teach, as well as hosted equipment). This will also be the time that the Mist cluster gets decommissioned.

'''Fri Sep 12 22:03:17 EDT 2025:''' HPSS software and OS upgrades are finished.

'''Tue Sep 9 17:05:38 EDT 2025:''' Starting tomorrow, Sep/10, and for the following 3 days HPSS will be down for software and OS upgrades. We will strive to finish sooner, at which time we will make the system available to users again.

===Mist/Niagara Decommissioning Schedule===

'''September 4, 2025'''
* Niagara reduced to 863 compute nodes.

'''September 9, 2025'''
* Niagara's Open OnDemand decommissioned.
* Brief data centre connection outage at 9 AM EDT
* Niagara reduced to 647 compute nodes at end of day.

'''September 11, 2025'''
* Trillium Open OnDemand goes live.

'''September 16, 2025'''
* '''Full-day data centre maintenance'''
* Niagara reduced to 431 compute nodes.
* Mist decommissioned.

'''September 24, 2025'''
* Niagara reduced to 215 compute nodes.

'''September 30, 2025'''
* Niagara decommissioned.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2025-09-17T00:32:37Z

Pinto:

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Down | Niagara|Niagara_Quickstart}}
|{{Down | Teach|Teach}}
|{{Down | Rouge|Rouge}}
|-
|{{Down | OnDemand|Open_OnDemand_Quickstart}}
|{{Down | Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up | File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up | External Network|Niagara_Quickstart#Logging_in}}
|{{Partial| Globus |Globus}}
|-
|{{Down | Balam|Balam}}
|{{Up | Cvmfs|Using_modules}}
|{{Down | Mist|Mist}}
|}

'''Tue Sep 16, 2025, 5:45 pm:''' Unfortunately, we cannot bring all systems up yet because we are waiting for a spare part for the cooling system that will be brought tomorrow. In the meantime, we will try bring Trillium up, but only the login nodes of other systems.

'''Tue Sep 16, 2025, from 7:00 am to 5:00 pm (EDT):''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Niagara, Mist, HPSS, Rouge, Teach, as well as hosted equipment). This will also be the time that the Mist cluster gets decommissioned.

'''Fri Sep 12 22:03:17 EDT 2025:''' HPSS software and OS upgrades are finished.

'''Tue Sep 9 17:05:38 EDT 2025:''' Starting tomorrow, Sep/10, and for the following 3 days HPSS will be down for software and OS upgrades. We will strive to finish sooner, at which time we will make the system available to users again.

===Mist/Niagara Decommissioning Schedule===

'''September 4, 2025'''
* Niagara reduced to 863 compute nodes.

'''September 9, 2025'''
* Niagara's Open OnDemand decommissioned.
* Brief data centre connection outage at 9 AM EDT
* Niagara reduced to 647 compute nodes at end of day.

'''September 11, 2025'''
* Trillium Open OnDemand goes live.

'''September 16, 2025'''
* '''Full-day data centre maintenance'''
* Niagara reduced to 431 compute nodes.
* Mist decommissioned.

'''September 24, 2025'''
* Niagara reduced to 215 compute nodes.

'''September 30, 2025'''
* Niagara decommissioned.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2025-09-16T23:14:03Z

Pinto: /* System Status */

__NOTOC__
{| style="border-spacing:10px; width: 95%"
| style="padding:1em; padding-top:.1em; border:2px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|

==System Status==


{|style="width:100%"
|{{Down3 | Trillium|https://docs.alliancecan.ca/wiki/Trillium_Quickstart}}
|{{Down | Niagara|Niagara_Quickstart}}
|{{Down | Teach|Teach}}
|{{Down | Rouge|Rouge}}
|-
|{{Down | OnDemand|Open_OnDemand_Quickstart}}
|{{Down | Scheduler|Niagara_Quickstart#Submitting_jobs}}
|{{Up | File system|Niagara_Quickstart#Storage_and_quotas}}
|{{Up | Burst Buffer|Burst_Buffer}}
|-
|{{Up | HPSS|HPSS}}
|{{Up | Login Nodes|Niagara_Quickstart#Logging_in}}
|{{Up | External Network|Niagara_Quickstart#Logging_in}}
|{{Down | Globus |Globus}}
|-
|{{Down | Balam|Balam}}
|{{Up | Cvmfs|Using_modules}}
|{{Down | Mist|Mist}}
|}

'''Tue Sep 16, 2025, 5:45 pm:''' Unfortunately, we cannot bring all systems up yet because we are waiting for a spare part for the cooling system that will be brought tomorrow. In the meantime, we will try bring Trillium up, but only the login nodes of other systems.

'''Tue Sep 16, 2025, from 7:00 am to 5:00 pm (EDT):''' The SciNet datacentre will undergo maintenance of several critical parts of the centre. This will require a full shutdown of all SciNet systems (Trillium, Niagara, Mist, HPSS, Rouge, Teach, as well as hosted equipment). This will also be the time that the Mist cluster gets decommissioned.

'''Fri Sep 12 22:03:17 EDT 2025:''' HPSS software and OS upgrades are finished.

'''Tue Sep 9 17:05:38 EDT 2025:''' Starting tomorrow, Sep/10, and for the following 3 days HPSS will be down for software and OS upgrades. We will strive to finish sooner, at which time we will make the system available to users again.

===Mist/Niagara Decommissioning Schedule===

'''September 4, 2025'''
* Niagara reduced to 863 compute nodes.

'''September 9, 2025'''
* Niagara's Open OnDemand decommissioned.
* Brief data centre connection outage at 9 AM EDT
* Niagara reduced to 647 compute nodes at end of day.

'''September 11, 2025'''
* Trillium Open OnDemand goes live.

'''September 16, 2025'''
* '''Full-day data centre maintenance'''
* Niagara reduced to 431 compute nodes.
* Mist decommissioned.

'''September 24, 2025'''
* Niagara reduced to 215 compute nodes.

'''September 30, 2025'''
* Niagara decommissioned.


[[Previous messages]]

{|style="border-spacing: 10px;width: 100%"
|valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== QuickStart Guides ==
* [https://docs.alliancecan.ca/wiki/Trillium_Quickstart Trillium Quickstart]
* [[Niagara Quickstart]]
* [[HPSS | HPSS archival storage]]
* [[Mist| Mist Power 9 GPU cluster]]
* [[Teach|Teach cluster]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging SciNet]]
| valign="top" style="margin: 1em; padding:1em; padding-top:.1em; border:2px solid #000; background-color:#fff; border-radius:7px; width: 49.5%" |

== Tutorials, Manuals, etc. ==
* [https://education.scinet.utoronto.ca SciNet education material]
* [https://www.youtube.com/c/SciNetHPCattheUniversityofToronto SciNet's YouTube channel]
* [[Modules specific to Niagara|Software Modules specific to Niagara]]
* [[Modules for Mist]]
* [[Commercial software]]
* [[Burst Buffer]]
* [[SSH#SSH Keys|SSH keys]]
* [[SSH Tunneling]]
* [[Visualization]]
* [[Running Serial Jobs on Niagara]]
* [[Jupyter Hub]]
|}

Main Page

2025-09-16T11:20:06Z

Pinto:

Main Page

2025-09-13T02:04:49Z

Pinto: