Difference between revisions of "Python"

From SciNet Users Documentation
Jump to: navigation, search
(Installing your own Python Modules)
Line 62: Line 62:
 
== Installing your own Python Modules ==
 
== Installing your own Python Modules ==
  
Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional  packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories.  This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.
+
If you need to install your own Python modules, either in Anaconda or in regular Python, you should set up a virtual environment (or a 'conda' environment if you're using Anaconda)Visit the [[PythonVirtualEnv]] page for instructions on how to set this up.
 
 
To install your own Python modules, follow the instructions below.  Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.
 
 
 
* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
 
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
 
<source lang=bash>
 
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
 
</source>
 
 
 
* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.
 
 
 
* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
 
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
 
<source lang=bash>
 
easy_install --prefix=${HOME} -O1 [packagename]
 
</source>
 
 
 
** Continue doing this until all of the packages you need to install are successfully installed.
 
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
 
<source lang=bash>
 
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
 
</source>
 
 
 
* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
 
** Download the relevant files
 
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
 
** cd into the newly created directory, and run
 
<source lang=bash>
 
python setup.py install --prefix=${HOME}
 
</source>
 
 
 
* Now, the install process may have added some .egg files or directories to your pathFor each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
 
<source lang=bash>
 
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
 
</source>
 
 
 
* You should now be done!  Now, re-source your .bashrc and test your new python modules.
 
 
 
* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own  modules (for the "module" system, not specifically python modules). 
 
 
 
[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.
 
 
 
  
 
== Producing Matplotlib Figures on Niagara Compute Nodes and in Job Scripts ==
 
== Producing Matplotlib Figures on Niagara Compute Nodes and in Job Scripts ==

Revision as of 18:38, 9 July 2018

Python is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the Python.org webpage; SciNet has given a mini-course of 8 lectures on Research Computing with Python in the Fall of 2013. An excellent set of material for teaching scientists to program in Python is also available at the Software Carpentry homepage.


Python on Niagara

We currently have three families of Python installed.

  • Anaconda
  • Intel Python
  • regular Python

Here we describe the differences between these packages.

Anaconda

Anaconda is a pre-assembled set of commonly-used self-consistent Python packages. The source for this collection is here. There are two types of Anaconda Python available:

  • The whole Anaconda software stack (the anacondaX/A.B.C modules).
  • Anaconda's Python, with all the Python packages, but without the rest of the Anaconda stack (gcc, bzip2, HDF5/NetCDF tools, etc) (the python/A.B.C-anacondaX.Y.Z modules).

As of 9 July 2018 the following Anaconda modules are available:

   $ module avail anaconda
   ----------------- /scinet/niagara/software/2018a/modules/base ------------------
    anaconda2/5.1.0    python/2.7.14-anaconda5.1.0    r/3.4.3-anaconda5.1.0
    anaconda3/5.1.0    python/3.6.4-anaconda5.1.0

Note that none of these modules require a compiler to be loaded. Also, note the presence of the R module. Anaconda now also comes with R; this package is the R analogy to the Anaconda Python modules.

You load the module in the usual way:

    $ module load anaconda3/5.1.0
    $ python
    >>>

Intel Python

The Intel Python modules are based on the Anaconda package. Intel has modified the package, and optimized the libraries to use the MKL libraries, which should make them faster than the Anaconda modules for some calculations.

As of 9 July 2018 the following Intel Python modules are available:

   $ module avail intelpython
   ----------------- /scinet/niagara/software/2018a/modules/base ------------------
    intelpython2/2018.2    intelpython3/2018.2

Regular Python

The base Python program has also been installed from source. This installation comes with no Python packages installed other than virtualenv and pip. You can use this module, in concert with virtualenv and pip, to build your own virtual environment.

    $ module avail python
    ----------------- /scinet/niagara/software/2018a/modules/base ------------------
    intelpython2/2018.2            python/2.7.14
    intelpython3/2018.2            python/3.6.4-anaconda5.1.0
    python/2.7.14-anaconda5.1.0    python/3.6.5               (D)
    $ module load python/3.6.5
    $ python
    >>>

Installing your own Python Modules

If you need to install your own Python modules, either in Anaconda or in regular Python, you should set up a virtual environment (or a 'conda' environment if you're using Anaconda). Visit the PythonVirtualEnv page for instructions on how to set this up.

Producing Matplotlib Figures on Niagara Compute Nodes and in Job Scripts

The conventional way of producing figures from python using matplotlib i.e.,

   import matplotlib.pyplot as plt
   plt.plot(.....)
   plt.savefig(...)

will not work on the Niagara compute nodes. The reason is that pyplot will try to open the figure in a window on the screen, but the compute nodes do not have screens or window managers. There is an easy workaround, however, that sets up a different 'backend' to matplotlib, one that does not try to open a window, as follows:

   import matplotlib as mpl
   mpl.use('Agg')
   import matplotlib.pyplot as plt
   plt.plot(.....)
   plt.savefig(...)

It is essential that the mpl.use('Agg') command precedes the importing of pyplot.