Using modules: Difference between revisions

From SciNet Users Documentation
Jump to navigation Jump to search
No edit summary
No edit summary
 
(2 intermediate revisions by the same user not shown)
Line 1: Line 1:
In the simplest case, the software you need will already be installed on Niagara or Mist. It will then be accessible in the form of a "module". If this is not the case, you can either ask SciNet staff to install it for you, or do it yourself.
Environment modules are documented on the Alliance wiki:
https://docs.alliancecan.ca/wiki/Using_modules


Modules are configuration files that contain instructions for modifying your software environment. This modular architecture allows multiple versions of the same application to be installed without conflict.  On the SciNet systems, modules are managed with the [https://www.tacc.utexas.edu/research-development/tacc-projects/lmod Lmod] tool developed at [https://www.tacc.utexas.edu TACC].
SciNet-specific:
 
* On Balam, the default software stack is <code>BalamEnv</code>, currently the 2023a epoch.  No other modules are loaded by default.  If it is not loaded, use <code>module load BalamEnv</code>; for GPU software, load <code>cuda</code> first.  See [[Balam#Software_stacks:_BalamEnv_and_CCEnv]].
A "modulefile" contains the information needed to make an application or library available in the user's login session. Typically a module file contains instructions that modify or initialize environment variables such as PATH and LD_LIBRARY_PATH in order to use different installed programs.  Note that the simple fact of loading a module doesn't execute the software in question.  To learn the name of the program binary or the syntax for its use, you should read the documentation for this software.  By using the module command, you shouldn't normally need to know the exact location or path of the software or library but you can nevertheless see such details about the module by means of the command <code>module show MODULENAME</code>.
* The Alliance <code>CCEnv</code>/<code>StdEnv/2023</code> stack is also available on Balam with <code>module load CCEnv StdEnv/2023</code>.
 
* The commercial module tree is at <code>/opt/software/commercial/modules</code>. See [[Commercial software]].
__TOC__
 
= Loading software modules =
 
The command <code>module spider</code> shows the available software.
 
For example:
<source lang="bash">
nia-login07:~$ module spider
---------------------------------------------------
The following is a list of the modules currently av
---------------------------------------------------
  CCEnv: CCEnv
 
  NiaEnv: NiaEnv/2018a
 
  anaconda2: anaconda2/5.1.0
 
  anaconda3: anaconda3/5.1.0
 
  autotools: autotools/2017
    autoconf, automake, and libtool
 
  boost: boost/1.66.0
 
  cfitsio: cfitsio/3.430
 
  cmake: cmake/3.10.2 cmake/3.10.3
 
  ...
</source>
 
The list contains the name of the software, followed by the available version.  Some software have their [[Software_pages | own pages]] describing what they are and how to use them.
 
== Module subcommands ==
 
Common module subcommands are:
 
* <code>module load <module-name></code>: load the default version of a particular software.
* <code>module load <module-name>/<module-version></code>: load a specific version of a particular software.
* <code>module purge</code>: unload all currently loaded modules.
* <code>module spider</code> (or <code>module spider <module-name></code>): list available software packages.
* <code>module avail</code>: list currently loadable software packages.
* <code>module list</code>: list loaded modules.
 
Along with modifying common environment variables, such as PATH, and LD_LIBRARY_PATH, these modules also create a SCINET_MODULENAME_ROOT environment variable, which can be used to access commonly needed software directories, such as $SCINET_MODULENAME_ROOT/include and $SCINET_MODULENAME_ROOT/lib.
 
== Finding experimental modules ==
When modules are first created, they are often put into an 'experimental' state until they are tested and declared to be worthy of use.  If you are looking for an experimental module, use the command
  module --show_hidden spider <module-name>
This will show the experimental modules, which can be loaded in the usual way.
 
== Software stacks: NiaEnv, MistEnv, and CCEnv ==
 
On Niagara, there are really two software stacks:
 
<ol style="list-style-type: decimal;">
<li><p>A [https://docs.scinet.utoronto.ca/index.php/Modules_specific_to_Niagara Niagara software stack] tuned and compiled for this machine. This stack is available by default, but if not, can be reloaded with</p>
<source lang="bash">module load NiaEnv</source></li>
<li><p>The same  [https://docs.alliancecan.ca/wiki/Modules software stack available on {{theAlliance}}'s General Purpose clusters] [https://docs.alliancecan.ca/wiki/Graham Graham], [https://docs.alliancecan.ca/wiki/Cedar Cedar], [https://docs.alliancecan.ca/wiki/Beluga Beluga], and [https://docs.alliancecan.ca/wiki/Narval Narval:</p>
<source lang="bash">module load CCEnv</source>
<p>If you want the same default modules loaded as on Beluga and Narval, then afterwards also <code>module load StdEnv</code>.</p>
<p>If you want the same default modules loaded as on Cedar and Graham, then afterwards also <code>module load arch/avx2 StdEnv/2020</code>.</p>
 
</li></ol>
 
Note: the <code>*Env</code> modules are '''''sticky'''''; remove them by <code>--force</code>. However, you will rarely need to do this.
 
To switch from NiaEnv to CCEnv do this:
<source lang="bash">
module load CCEnv
module load StdEnv
module spider
... and so on ...
</source>
 
Occasionally we add an epoch to the NiaEnv stack with newer versions of the module.
 
On Mist, the modules are part of the [[Modules for Mist || MistEnv stack]], which also has several epochs.  Because Mist has an architecture which is not supported by the {{Alliance}} stack, CCEnv modules are not on that system.
 
 
== Saving combinations of modules ==
 
The module command can be used to save combinations of loaded modules.  Once you have an ideal combination of modules loaded, that you wish to save, you can use the commmand
  module save MYMODULES
to save the combination of modules, where MYMODULES is the name you wish to give this combination.  After you have saved your set of modules you can put the command
  module restore MYMODULES
in your job submission script to load this set of modules, or you can just type this command at the command line prompt.
 
To list all of your saved combinations of modules, use the command
  module savelist
To list which modules are saved in a particular module set, use the command
  module describe MYMODULES
 
== Tips for loading software ==
 
* We advise '''''against''''' loading modules in your .bashrc.  This can lead to very confusing behaviour under certain circumstances.  Our guidelines for .bashrc files can be found [[bashrc guidelines|here]].
* The default .bashrc and .bash_profile files on Niagara and Mist can be found [[bashrc guidelines|here]].
* Instead, load modules by hand when needed, or by sourcing a separate script.
* Load run-specific modules inside your job submission script.
* Short names give default versions; e.g. <code>intel</code> → <code>intel/2018.2</code>. It is usually better to be explicit about the versions, for future reproducibility.
* Handy abbreviations:
  ml → module list
  ml NAME → module load NAME  # if NAME is an existing module
  ml X → module X
* Modules sometimes require other modules to be loaded first.<br />
Solve these dependencies by using <code>module spider</code>.
 
= Module spider =
 
Oddly named, the module subcommand spider is the search-and-advice facility for modules.
 
Suppose one wanted to load the openmpi module. Upon trying to load the module, one may get the following message:
<source lang="bash">
nia-login07:~$ module load openmpi
Lmod has detected the error:  These module(s) exist but cannot be loaded as requested: "openmpi"
  Try: "module spider openmpi" to see how to load the module(s).
</source>
So while that fails, following the advice that the command outputs, the next command would be:
<source lang="bash">
nia-login07:~$ module spider openmpi
------------------------------------------------------------------------------------------------------
  openmpi:
------------------------------------------------------------------------------------------------------
    Versions:
        openmpi/2.1.3
        openmpi/3.0.1
        openmpi/3.1.0
 
------------------------------------------------------------------------------------------------------
  For detailed information about a specific "openmpi" module (including how to load the modules) use
  the module s full name.
  For example:
 
    $ module spider openmpi/3.1.0
------------------------------------------------------------------------------------------------------
</source>
So this gives just more detailed suggestions on using the spider command. Following the advice again, one would type:
<source lang="bash">
nia-login07:~$ module spider openmpi/3.1.0
------------------------------------------------------------------------------------------------------
  openmpi: openmpi/3.1.0
------------------------------------------------------------------------------------------------------
    You will need to load all module(s) on any one of the lines below before the "openmpi/3.1.0"
    module is available to load.
 
      NiaEnv/2018a  gcc/7.3.0
      NiaEnv/2018a  intel/2018.2
</source>
These are concrete instructions on how to load this particular openmpi module. Following these leads to a successful loading of the module.
<source lang="bash">
nia-login07:~$ module load NiaEnv/2018a  intel/2018.2  # note: NiaEnv is usually already loaded
nia-login07:~$ module load openmpi/3.1.0
</source>
<source lang="bash">
nia-login07:~$ module list
Currently Loaded Modules:
  1) NiaEnv/2018a (S)  2) intel/2018.2  3) openmpi/3.1.0
 
  Where:
  S:  Module is Sticky, requires --force to unload or purge</source>
 
= Creating custom modules =
I you desire to create your own modules to use on Niagara or Mist, perform the following steps:
* Create a directory to hold you modules, say ~/mymodules.
* Create and edit a module file, say ~/mymodules/foo/version (see for example /scinet/niagara/software/2018a/modules/base/gcc/7.3.0)
* Run the command: <code>module use ~/mymodules</code>.
* Load your module: <code>module load foo/version</code>.

Latest revision as of 18:26, 9 November 2026

Environment modules are documented on the Alliance wiki: https://docs.alliancecan.ca/wiki/Using_modules

SciNet-specific:

  • On Balam, the default software stack is BalamEnv, currently the 2023a epoch. No other modules are loaded by default. If it is not loaded, use module load BalamEnv; for GPU software, load cuda first. See Balam#Software_stacks:_BalamEnv_and_CCEnv.
  • The Alliance CCEnv/StdEnv/2023 stack is also available on Balam with module load CCEnv StdEnv/2023.
  • The commercial module tree is at /opt/software/commercial/modules. See Commercial software.