Difference between revisions of "SciNet Command Line Utilities"

From SciNet Users Documentation
Jump to: navigation, search
(Storage)
(Jobs & Queues)
 
(10 intermediate revisions by one other user not shown)
Line 21: Line 21:
 
|''-h'' to see all command options
 
|''-h'' to see all command options
 
|Informs about the user and group file system usage.
 
|Informs about the user and group file system usage.
|Niagara
+
|Niagara, BGQ
 
|-
 
|-
 
|topUserDirOver1000list
 
|topUserDirOver1000list
Line 36: Line 36:
  
 
= Jobs & Queues =
 
= Jobs & Queues =
{| border="1" cellpadding="5" cellspacing="0"
+
{| class = "wikitable"
!{{Hl2}} | Command
+
!Command
!{{Hl2}}| Arguments
+
!Arguments
!{{Hl2}}| Description
+
!Description
!{{Hl2}}| Cluster
+
!Cluster
 
|-
 
|-
 
|qsum
 
|qsum
|most showq arguments work
+
|most squeue arguments work
 
|Lists jobs running or in the queue, grouped by user.
 
|Lists jobs running or in the queue, grouped by user.
|GPC,TCS
+
|Niagara
 
|-
 
|-
|llq1
+
|llq2
 
|
 
|
 
|Detailed information on jobs that are actively running.
 
|Detailed information on jobs that are actively running.
|TCS
+
|BGQ
 
|-
 
|-
|scinet gpc priority
+
|scinet niagara priority
 
|
 
|
|Compute usage in the last 14 days, and how this affects your priority in the queue.
+
|Compute usage in the last 7 days, and how this affects your priority in the queue.
|GPC
+
|Niagara
 
|-
 
|-
 
|nodeperf
 
|nodeperf
 
|[''userName'']
 
|[''userName'']
 
|Who is doing what on the current node?
 
|Who is doing what on the current node?
|GPC
+
|Niagara
 
|-
 
|-
|jobperf
+
|[[Slurm#jobperf | jobperf]]
 
| <''jobId''<nowiki> | </nowiki>''jobName''>
 
| <''jobId''<nowiki> | </nowiki>''jobName''>
 
|Informs about the performance of all nodes of a given job.
 
|Informs about the performance of all nodes of a given job.
|GPC
+
|Niagara
 
|-
 
|-
|jobError
+
|[[Slurm#Debug | debugjob]]
| <''jobId''<nowiki> | </nowiki>''jobName''>
+
|[''number of nodes'']
| Displays realtime error output of a given job.
+
|Requests a time-limited interactive session of up to 4 dedicated nodes
|GPC
+
|Niagara
|-
 
|jobOutput
 
| <''jobId''<nowiki> | </nowiki>''jobName''>
 
| Displays realtime standard output of a given job.
 
|GPC
 
|-
 
|jobcd
 
| <''jobId''<nowiki> | </nowiki>''jobName''>
 
| Allows users to quickly move into the working directory of a given job.
 
|GPC
 
|-
 
|jobscript
 
| <''jobId''<nowiki> | </nowiki>''jobName''>
 
| Displays the submission script used when submitting a given job.
 
|GPC
 
|-
 
|jobssh
 
| <''jobId''<nowiki> | </nowiki>''jobName''>
 
| Allows users to connect to the head-node of a given job.
 
|GPC
 
|-
 
|jobtop
 
| <''jobId''<nowiki> | </nowiki>''jobName''>
 
| Allows users to "top" on the head-node of a given job.
 
|GPC
 
|-
 
|jobdep
 
|<''jobId''>
 
|Displays dependencies of given job.
 
|GPC
 
|-
 
|jobtree
 
|[''userName'']
 
|Displays the jobs tree of dependencies for a given user.
 
|GPC
 
|-
 
|debugjob
 
|[''nodes'']
 
|Requests a time-limited interactive session of up to 8 dedicated nodes
 
|GPC
 
 
|-
 
|-
 
|}
 
|}
  
 
= Modules =
 
= Modules =
{| border="1" cellpadding="5" cellspacing="0"
+
{|class = "wikitable"
!{{Hl2}} | Command
+
!|Command
!{{Hl2}}| Arguments
+
!| Arguments
!{{Hl2}}| Description
+
!| Description
!{{Hl2}}| Cluster
+
!| Cluster
 
|-
 
|-
|modulefind
+
|ml
|[''searchWord'']
+
|
|Finds modules containing ''searchWord'' and indicate their compatibility with already loaded modules.
+
|"module list"
|GPC
+
|Niagara
 
|-
 
|-
|moduleadvice
+
|ml
|<''modulename''>
+
|<modulename>
|Show the module commands required to load ''modulename'', taking into account prerequisites and conflicts with already loaded modules.
+
|"module load <modulename>"
|GPC
+
|Niagara
 
|-
 
|-
|modulewrap
+
|ml
|''<comma-separated-modules>'' ''<command>''
+
|X
|Load the modules in the ''comma-separated-modules'' list (no spaces!) only to run the given ''command''.
+
|"module X"
|GPC
+
|Niagara
 
|-
 
|-
 
|}
 
|}
 
  
 
= Related Topics =
 
= Related Topics =
[https://wiki.scinet.utoronto.ca/wiki/index.php/Moab#Checking_the_memory_usage_from_jobs Checking the memory usage from jobs]
+
[[Slurm#Monitoring_jobs | Monitoring jobs]]
 
<br>
 
<br>
[https://wiki.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart GPC Quickstart]
+
[[Niagara_Quickstart | Niagara Quickstart]]
 
<br>
 
<br>
[https://wiki.scinet.utoronto.ca/wiki/index.php/Moab Scheduler]
+
[[Slurm | Slurm Scheduler]]
 
<br>
 
<br>
[https://wiki.scinet.utoronto.ca/wiki/index.php/FAQ#Monitoring_jobs_in_the_queue Monitoring Jobs]
+
[[FAQ#Monitoring_jobs_in_the_queue | Monitoring Jobs]]
 
<br>
 
<br>
 
[https://wiki.scinet.utoronto.ca/wiki/images/a/a0/TechTalkJobMonitoring.pdf Tech Talk on Monitoring Jobs]
 
[https://wiki.scinet.utoronto.ca/wiki/images/a/a0/TechTalkJobMonitoring.pdf Tech Talk on Monitoring Jobs]

Latest revision as of 15:02, 16 August 2018

Below is list of handy command-line utilities written by SciNet to help manage your data and compute.

Note that in the table, optional arguments are denoted with square brackets (i.e. '[]'), while mandatory arguments are denoted with angular brackets (i.e. '<>').

Storage

Command Arguments Description Cluster
quota Short overview of a user's storage usage. Niagara
diskUsage -h to see all command options Informs about the user and group file system usage. Niagara, BGQ
topUserDirOver1000list Lists your directories that have over 1,000 files Niagara
topUserDirOver1GBlist Lists your directories that have over 1 GB of data Niagara

Jobs & Queues

Command Arguments Description Cluster
qsum most squeue arguments work Lists jobs running or in the queue, grouped by user. Niagara
llq2 Detailed information on jobs that are actively running. BGQ
scinet niagara priority Compute usage in the last 7 days, and how this affects your priority in the queue. Niagara
nodeperf [userName] Who is doing what on the current node? Niagara
jobperf <jobId | jobName> Informs about the performance of all nodes of a given job. Niagara
debugjob [number of nodes] Requests a time-limited interactive session of up to 4 dedicated nodes Niagara

Modules

Command Arguments Description Cluster
ml "module list" Niagara
ml <modulename> "module load <modulename>" Niagara
ml X "module X" Niagara

Related Topics

Monitoring jobs
Niagara Quickstart
Slurm Scheduler
Monitoring Jobs
Tech Talk on Monitoring Jobs