STAR-CCM+

The STAR-CCM+ engineering simulation tool is installed on the CC software stack.

Getting a license

Licences are provided by Siemens.

You will need to create a $HOME/.licenses/starccm.lic file with the following contents:

SERVER 127.0.0.1 ANY 1999
USE_SERVER

and establish the ssh tunnel to the Siemens licence server as shown in the submission script template further below. Also be sure to define the licence env vars prior to loading the modules.

Running using the CC installation

STAR-CCM+ 21.02.008

STAR-CCM+ is only available on the CC software stack. On Trillium, load the module with:

module load starccm/21.02.008-R8

Other versions may be available; check with module spider starccm. If you use a different version, change the .star- directory in the next section to match it.

Setting up your .star-21.02.008 directory

STAR-CCM+ version 21.02.008 will attempt to write to your $HOME/.star-21.02.008 directory. This will work when you are testing your workflow on the login nodes, because they can write to $HOME. However, recall that the compute nodes cannot write to the /home filesystem. If you attempt to run STAR-CCM+ from a compute node using the default configuration, it will fail because STAR-CCM+ cannot write to $HOME/.star-21.02.008.

The solution is to create an alternative directory called $SCRATCH/.star-21.02.008, and create a soft link from $SCRATCH/.star-21.02.008 to $HOME/.star-21.02.008:

mkdir $SCRATCH/.star-21.02.008
ln -s $SCRATCH/.star-21.02.008 $HOME/.star-21.02.008

This makes STAR-CCM+ think it is writing to $HOME/.star-21.02.008, when in fact it is writing to $SCRATCH/.star-21.02.008. This command only needs to be run once, though it needs to be run for each version of STAR-CCM+ you run.

Running STAR-CCM+

Example submission script for a job running on 2 nodes, with maximum walltime of 30 minutes:

#!/bin/bash
#SBATCH --time=0-00:30        # Time limit: d-hh:mm
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=40
#SBATCH --mail-type=BEGIN
#SBATCH --mail-type=END

cd $SLURM_SUBMIT_DIR

ssh tri-gw -L 1999:flex.cd-adapco.com:1999 -L 2099:flex.cd-adapco.com:2099 -N -f
export LM_PROJECT='YOUR CD-ADAPCO PROJECT ID GOES HERE'
export CDLMD_LICENSE_FILE="1999@127.0.0.1"

module load starccm/21.02.008-R8

slurm_hl2hl.py --format STAR-CCM+ > $SLURM_SUBMIT_DIR/machinefile_$SLURM_JOB_ID

NCORE=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

# Workaround for license failures
# Try up to 5 times to get starccm+ to start by checking exit status (throws 143 when fails, 0 when works).
i=1
RET=-1
while [ $i -le 5 ] && [ $RET -ne 0 ]; do
        [ $i -eq 1 ] || sleep 5
        echo "Attempt number: "$i
        starccm+ -power -np $NCORE -podkey $LM_PROJECT   -machinefile $SLURM_SUBMIT_DIR/machinefile_$SLURM_JOB_ID -batch $SLURM_SUBMIT_DIR/your-simulation-file.java $SLURM_SUBMIT_DIR/your-simulation-file.sim > $SLURM_JOB_ID.results
        RET=$?
        i=$((i+1))
   done
exit $RET

Licence debugging

After establishing the ssh tunnel, you may check that you can reach the server from the compute/debug node with:

$ telnet localhost 1999

If there is no response there is a problem with the tunnel or the server at flex.cd-adapco.com

You may debug the license acquisition process on a compute/debug node (you may use on your script as well) by issuing the following:

$ lmutil lmstat -c 1999@localhost

To check all the licenses available you may use:

$ lmutil lmstat -c 1999@localhost -a

To check the license server from a login node you may use:

lmutil lmstat -c 1999@flex.cd-adapco.com

If you see the exception below on the output of your run, you need to ask your PI to renew the podkey/license for your group

Asked for 1 license of ccmppower and got 0
Asked for 1 license of ccmppowerplus and got 0
PoD database message: -20002 *FAILED_TOKEN_EXPIRED*

If you are using your own license server, you will need to ask its administrator to allow incoming STAR-CCM+ license requests from 142.150.188.58, the outgoing address of the SciNet tunnel gateway tri-gw. This applies to jobs on both Trillium and Balam; the login nodes are not on the licence path.

See SSH tunnelling for background on the gateway.