HybridX on P7

You can have two different approaches when using HybridX on P7 cluster.

1. Using OpenMPI
2. Using PE - Parallel environment from IBM

It has been observed that OpenMPI runs only on a single node due to an InfiniBand issue. If you plan to use 1 node jobs you can follow the instructions below:

How to compile HybridX using OpenMPI

Following script assumes HybridX code is located under "$HOME/HybridCode". It compiles the package using GCC-4.8 and OpenMPI-1.6.5 and installs it to the "build-p7/install" directory inside the HybridX. See script below and modify if you want make changes:

#!/bin/sh

# Load modules
module load gcc/4.8.1 cmake/2.8.8 openmpi/1.6.5-gcc

# Package details
base=/$HOME/HybridCode
pkg=HybridX

cd $base/$pkg

# Variables for installation
src=$base/$pkg
bld=$base/$pkg/build-p7

# Start from scratch each time when this script is executed
rm -rf $bld
mkdir -p $bld
cd $bld

# Run cmake
cmake $src
# cmake -DBOOST_ROOT=${SCINET_BOOST_DIR} $src

# Compile and install
gmake
gmake install

When the compilation is completed successfully, you should be able to find HybridX executable under the "$HOME/HybridCode/HybridX/build-p7/install/bin". Using the binary executable, you can run HybridX simulations. Please see following job script:

#!/bin/bash
##===================================
## P7 Load Leveler Submission Script
##===================================
##
## Don't change these parameters unless you really know what you are doing
##
##@ environment = MP_INFOLEVEL=0; MP_USE_BULK_XFER=yes; MP_BULK_MIN_MSG_SIZE=64K; \
##                MP_EAGER_LIMIT=64K; MP_DEBUG_ENABLE_AFFINITY=no
##
##===================================
## Avoid core dumps
## @ core_limit   = 0
##===================================
## Job specific
##===================================
#
# @ job_name = hybridx-isotropic
# @ job_type = parallel
# @ class = verylong
# @ output = $(jobid).out
# @ error = $(jobid).err
# @ wall_clock_limit = 01:00:00
# @ node = 1
# @ tasks_per_node = 128
# @ queue
#
#===================================

# Load modules
module purge
module load gcc/4.8.1
module load openmpi/1.6.5-gcc

# HybridX folders
export hybrid_root=$HOME/HybridCode/HybridX/build-p7/install
export hybrid_bin=${hybrid_root}/bin
export hybrid_run=$HOME/HybridCode/run

# Go to case folder
cd $hybrid_run/isotropic-p7

mpirun -np 128 ${hybrid_bin}/Hybrid -i isotropic.input 2>&1 | tee log.hybridx.isotropic

How to compile HybridX using PE

This is very similar to using OpenMPI except a few things. Make the following modifications to your compilation script given above:

#!/bin/sh

# Load modules
module load gcc/4.8.1 cmake/2.8.8 pe/1.2.0.9

bld=$base/$pkg/build-p7-pe

# For parallel environment on P7
export CXXFLAGS="-cpp"
export LDFLAGS="-Wl,--allow-multiple-definition"

Rest of the compilation should be the same as before. If you can compile successfully, you can use the job script:

#!/bin/bash
##===================================
## P7 Load Leveler Submission Script
##===================================
##
## Don't change these parameters unless you really know what you are doing
##
##@ environment = MP_INFOLEVEL=0; MP_USE_BULK_XFER=yes; MP_BULK_MIN_MSG_SIZE=64K; \
##                MP_EAGER_LIMIT=64K; MP_DEBUG_ENABLE_AFFINITY=no
##
##===================================
## Avoid core dumps
## @ core_limit   = 0
##===================================
## Job specific
##===================================
#
# @ job_name = hybridx-isotropic
# @ job_type = parallel
# @ class = verylong
# @ output = $(jobid).out
# @ error = $(jobid).err
# @ wall_clock_limit = 01:00:00
# @ node = 4
# @ tasks_per_node = 32
# @ queue
#
#===================================

# Load modules
module purge
module load gcc/4.8.1 pe/1.2.0.9

# HybridX folders
export hybrid_root=$HOME/HybridCode/HybridX/build-p7-pe/install
export hybrid_bin=${hybrid_root}/bin
export hybrid_run=$HOME/HybridCode/run

# Go to case folder
cd $hybrid_run/isotropic-p7

mpiexec -n 128 ${hybrid_bin}/Hybrid -i isotropic.input 2>&1 | tee log.hybridx.isotropic

Notes:

1. P7 processors support 4 threads per core, so you can increase the number of tasks accordingly.
2. Scheduler is the same as Blue Gene, LoadLeveler so same commands apply. llq, llsubmit etc.
3. LoadLeveler writes results to an output file specified in the job details so you don't need the tee command given in the example above.
4. P7 cluster share the same file-system with Blue Gene so be careful with that.