Difference between revisions of "Globus"

From SciNet Users Documentation
Jump to: navigation, search
 
Line 3: Line 3:
 
__TOC__
 
__TOC__
  
=== Copying files between Compute Canada sites using the Globus web interface ===
+
== Copying files between Compute Canada sites using the Globus web interface ==
  
 
The procedure of using the [http://globus.computecanada.ca/ Globus web interface] for transferring data between different Compute Canada sites ("endpoints" in Globus), including those at SciNet, is well described in the [https://docs.computecanada.ca/wiki/Globus Compute Canada Globus documentation].
 
The procedure of using the [http://globus.computecanada.ca/ Globus web interface] for transferring data between different Compute Canada sites ("endpoints" in Globus), including those at SciNet, is well described in the [https://docs.computecanada.ca/wiki/Globus Compute Canada Globus documentation].
Line 9: Line 9:
 
At SciNet, there are three endpoints:
 
At SciNet, there are three endpoints:
  
====<tt>computecandada#niagara</tt>====
+
=== <tt>computecandada#niagara</tt> ===
 +
 
 
This endpoint gives access to the files on your Niagara $HOME.  <br>To get easy access to your files on your $SCRATCH in the Globus web interface, it can be helpful to add a so-called softlink in your home directory to the scratch directory, by issuing the following command on the command line on Niagara (once):<br/><tt> $ ln -sn $SCRATCH $HOME/scratch</tt><br/>A similar soft-link can be useful for your $PROJECT directory, if you have one on Niagara.
 
This endpoint gives access to the files on your Niagara $HOME.  <br>To get easy access to your files on your $SCRATCH in the Globus web interface, it can be helpful to add a so-called softlink in your home directory to the scratch directory, by issuing the following command on the command line on Niagara (once):<br/><tt> $ ln -sn $SCRATCH $HOME/scratch</tt><br/>A similar soft-link can be useful for your $PROJECT directory, if you have one on Niagara.
  
====<tt>computecanada#hpss</tt>====
+
=== <tt>computecanada#hpss</tt> ===
 +
 
 
This endpoint gives access to your [[HPSS]] space, if you have access to it.  [[HPSS]] is a tape-backed hierarchical storage system that provides a significant portion of the allocated storage space at SciNet.  The Globus endpoint is not the only way to interact with HPSS, and may not be appropriate method for your use case. Users should read the [[HPSS]] page before using this endpoint.  
 
This endpoint gives access to your [[HPSS]] space, if you have access to it.  [[HPSS]] is a tape-backed hierarchical storage system that provides a significant portion of the allocated storage space at SciNet.  The Globus endpoint is not the only way to interact with HPSS, and may not be appropriate method for your use case. Users should read the [[HPSS]] page before using this endpoint.  
  
====<tt>computecanada#bgq</tt>====
+
=== <tt>computecanada#bgq</tt> ===
 +
 
 
This endpoint gives access to the files on your SciNet SOSCIP $HOME, if you have access to one of the SOSCIP systems hosted at SciNet, i.e., the [[BGQ]] or the [[SOSCIP GPU]] cluster.  To get easy access in the Globus web interface to your SOSCIP $SCRATCH files, it can be helpful to add a so-called softlink in your home directory to your scratch directory, by issuing the following command on the command line on Niagara (once):<br/><tt> $ ln -sn $SCRATCH $HOME/scratch</tt><br/>
 
This endpoint gives access to the files on your SciNet SOSCIP $HOME, if you have access to one of the SOSCIP systems hosted at SciNet, i.e., the [[BGQ]] or the [[SOSCIP GPU]] cluster.  To get easy access in the Globus web interface to your SOSCIP $SCRATCH files, it can be helpful to add a so-called softlink in your home directory to your scratch directory, by issuing the following command on the command line on Niagara (once):<br/><tt> $ ln -sn $SCRATCH $HOME/scratch</tt><br/>
  
=== Copying files between Compute Canada sites and your personal computer using the Globus web interface===
+
== Copying files between Compute Canada sites and your personal computer using the Globus web interface ==
  
 
The procedure of using the [http://globus.computecanada.ca/ Globus web interface] for transferring data between Compute Canada sites (including those at SciNet) and your own personal computer, is well described in the [https://docs.computecanada.ca/wiki/Globus Compute Canada Globus documentation].  
 
The procedure of using the [http://globus.computecanada.ca/ Globus web interface] for transferring data between Compute Canada sites (including those at SciNet) and your own personal computer, is well described in the [https://docs.computecanada.ca/wiki/Globus Compute Canada Globus documentation].  
Line 24: Line 27:
 
Essentially, you create an "endpoint for your personal computer", then you can use the web interface to transfer to one of the SciNet endpoints listed above.
 
Essentially, you create an "endpoint for your personal computer", then you can use the web interface to transfer to one of the SciNet endpoints listed above.
  
=== Copying Files to Niagara From the Linux Command-line ===
+
== Copying Files to Niagara From the Linux Command-line ==
  
==== Step 1: Install globus CLI ====
+
=== Step 1: Install globus CLI ===
  
 
Requires python 2.7. On the machine you're transferring from:
 
Requires python 2.7. On the machine you're transferring from:
Line 34: Line 37:
 
  $ pip install globus-cli
 
  $ pip install globus-cli
  
==== Step 2: Login to globus ====
+
=== Step 2: Login to globus ===
  
 
  $ globus login
 
  $ globus login
Line 49: Line 52:
 
Visit the URL in a web browser, choose "Compute Canada" as your organization, and enter your Niagara username/password.
 
Visit the URL in a web browser, choose "Compute Canada" as your organization, and enter your Niagara username/password.
  
==== Step 3: Create a personal endpoint ====
+
=== Step 3: Create a personal endpoint ===
  
 
  $ globus endpoint create --personal my-endpoint-name
 
  $ globus endpoint create --personal my-endpoint-name
Line 63: Line 66:
 
Save this info as we'll need it later.
 
Save this info as we'll need it later.
  
==== Step 4: Get Globus Connect Personal ====
+
=== Step 4: Get Globus Connect Personal ===
  
 
  $ wget https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz
 
  $ wget https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz
Line 69: Line 72:
 
  $ cd globusconnectpersonal-x.y.z
 
  $ cd globusconnectpersonal-x.y.z
  
==== Step 5: Setup your endpoint ====
+
=== Step 5: Setup your endpoint ===
  
 
  $ ./globusconnectpersonal -setup yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy
 
  $ ./globusconnectpersonal -setup yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy
Line 79: Line 82:
 
  Done!
 
  Done!
  
==== Step 6: Configure your endpoint (optional) ====
+
=== Step 6: Configure your endpoint (optional) ===
  
 
By default globus only allows transfers to/from your home directory.
 
By default globus only allows transfers to/from your home directory.
Line 86: Line 89:
 
  /path/to/data/,0,1
 
  /path/to/data/,0,1
  
==== Step 7: Start Globus Connect ====
+
=== Step 7: Start Globus Connect ===
  
 
  $ ./globusconnectpersonal -start &
 
  $ ./globusconnectpersonal -start &
  
==== Step 8: Set some convenience variables ====
+
=== Step 8: Set some convenience variables ===
  
 
  $ my_endpoint="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
 
  $ my_endpoint="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
Line 97: Line 100:
 
replacing "xxxx..." with the Endpoint ID from 'globus endpoint create'.
 
replacing "xxxx..." with the Endpoint ID from 'globus endpoint create'.
  
==== Step 9: Activate the Niagara endpoint ====
+
=== Step 9: Activate the Niagara endpoint ===
  
 
  $ globus endpoint activate --myproxy --myproxy-lifetime=1000 $niagara_endpoint
 
  $ globus endpoint activate --myproxy --myproxy-lifetime=1000 $niagara_endpoint
  
==== Step 10: Start a transfer ====
+
=== Step 10: Start a transfer ===
  
 
  $ globus transfer --recursive $my_endpoint:/path/to/data $niagara_endpoint:/scratch/g/group/username/data
 
  $ globus transfer --recursive $my_endpoint:/path/to/data $niagara_endpoint:/scratch/g/group/username/data

Latest revision as of 20:00, 21 November 2019

Globus is a service for fast, reliable, secure data movement. Designed specifically for researchers, Globus has an easy-to-use interface with background monitoring features that automate the management of file transfers between any two resources, whether they are at Compute Canada, another supercomputing facility, a campus cluster, lab server, desktop or laptop.

Copying files between Compute Canada sites using the Globus web interface

The procedure of using the Globus web interface for transferring data between different Compute Canada sites ("endpoints" in Globus), including those at SciNet, is well described in the Compute Canada Globus documentation.

At SciNet, there are three endpoints:

computecandada#niagara

This endpoint gives access to the files on your Niagara $HOME.
To get easy access to your files on your $SCRATCH in the Globus web interface, it can be helpful to add a so-called softlink in your home directory to the scratch directory, by issuing the following command on the command line on Niagara (once):
$ ln -sn $SCRATCH $HOME/scratch
A similar soft-link can be useful for your $PROJECT directory, if you have one on Niagara.

computecanada#hpss

This endpoint gives access to your HPSS space, if you have access to it. HPSS is a tape-backed hierarchical storage system that provides a significant portion of the allocated storage space at SciNet. The Globus endpoint is not the only way to interact with HPSS, and may not be appropriate method for your use case. Users should read the HPSS page before using this endpoint.

computecanada#bgq

This endpoint gives access to the files on your SciNet SOSCIP $HOME, if you have access to one of the SOSCIP systems hosted at SciNet, i.e., the BGQ or the SOSCIP GPU cluster. To get easy access in the Globus web interface to your SOSCIP $SCRATCH files, it can be helpful to add a so-called softlink in your home directory to your scratch directory, by issuing the following command on the command line on Niagara (once):
$ ln -sn $SCRATCH $HOME/scratch

Copying files between Compute Canada sites and your personal computer using the Globus web interface

The procedure of using the Globus web interface for transferring data between Compute Canada sites (including those at SciNet) and your own personal computer, is well described in the Compute Canada Globus documentation.

Essentially, you create an "endpoint for your personal computer", then you can use the web interface to transfer to one of the SciNet endpoints listed above.

Copying Files to Niagara From the Linux Command-line

Step 1: Install globus CLI

Requires python 2.7. On the machine you're transferring from:

$ virtualenv venv-globus
$ source ./venv-globus/bin/activate
$ pip install globus-cli

Step 2: Login to globus

$ globus login

You should see:

Please log into Globus here:
---------------------------
https://auth.globus.org/v2/oauth2/...
---------------------------

Enter the resulting Authorization Code here:

Visit the URL in a web browser, choose "Compute Canada" as your organization, and enter your Niagara username/password.

Step 3: Create a personal endpoint

$ globus endpoint create --personal my-endpoint-name

Replace my-endpoint-name with a name of your choice.

You should see:

Message:     Endpoint created successfully
Endpoint ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Setup Key:   yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy

Save this info as we'll need it later.

Step 4: Get Globus Connect Personal

$ wget https://downloads.globus.org/globus-connect-personal/linux/stable/globusconnectpersonal-latest.tgz
$ tar -xzf globusconnectpersonal-latest.tgz
$ cd globusconnectpersonal-x.y.z

Step 5: Setup your endpoint

$ ./globusconnectpersonal -setup yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy

replacing "yyyy..." with the Setup Key from 'globus endpoint create'. You should see something like:

Configuration directory: /home/username/.globusonline/lta
Contacting relay.globusonline.org:2223
Done!

Step 6: Configure your endpoint (optional)

By default globus only allows transfers to/from your home directory. Edit ~/.globusonline/lta/config-paths and add a line for any other directories you need, e.g.:

/path/to/data/,0,1

Step 7: Start Globus Connect

$ ./globusconnectpersonal -start &

Step 8: Set some convenience variables

$ my_endpoint="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
$ niagara_endpoint="77506016-4a51-11e8-8f88-0a6d4e044368"

replacing "xxxx..." with the Endpoint ID from 'globus endpoint create'.

Step 9: Activate the Niagara endpoint

$ globus endpoint activate --myproxy --myproxy-lifetime=1000 $niagara_endpoint

Step 10: Start a transfer

$ globus transfer --recursive $my_endpoint:/path/to/data $niagara_endpoint:/scratch/g/group/username/data