Logo of Science Foundation Ireland  Logo of the Higher Education Authority, Ireland7 Capacities
Ireland's High-Performance Computing Centre | ICHEC
Home | News | Infrastructure | Outreach | Services | Research | Support | Education & Training | Consultancy | About Us | Login

ICHEC Software

Information about software packages installed on the ICHEC systems.

OpenFOAM

Versions Installed

Fionn: 2.2.2 / 2.4.0 / 3.0.1 / 4.0 / 4.1

Description

OpenFOAM is an open source package for finite-volume based solution of problems in continuum mechanics, primarily Computational Fluid Dynamics (CFD). OpenFOAM is supplied with several standard solvers for different classes of flow problems, including multiphase flow and combustion. The object-oriented C++ implementation of OpenFOAM facilitates the development of custom flow solvers.

License

OpenFOAM is available under the terms of the GNU General Public License.

Benchmarks

The OpenFOAM "cavity" tutorial case was scaled-up for benchmarking purposes by changing the number of points (in constant/polyMesh/blockMeshDict) from (20 20 1) to (512 512 1). The time-step deltaT (in system/controlDict) was reduced from 0.5 to 0.0001, and the endTime reduced to 0.01. A system/decomposeParDict file was introduced (essentially copied from wingMotion2D_pimpleDyMFoam) to control parallel decomposition.

The figure below shows total run-time for this case on Fionn (as well as our old system Stokes) as a function of core-count.

Job Submission Example

Like other jobs on ICHEC systems, OpenFOAM jobs must be submitted using a PBS script. The following is an example PBS script for allocating 1 node of Fionn (24 cores) for 30 minutes, then running the motorBike tutorial in parallel over 6 cores within that allocation. Note that OpenFOAM employs domain decomposition for parallel runs which is defined geometrically by decomposeParDict in the system directory. Each processor must be assigned a portion of the mesh on which to operate. For the case below, the corresponding decomposeParDict, the default shipped with the source, defines 6 regions. The solvers can therefore only be run over 6 cores. To vary this, the decomposeParDict must be edited accordingly (both numberOfSubdomains, and the (nx ny nz) "n" coeffs). More detailed information on running OpenFOAM in parallel can be found here.

Before running the PBS script, we first locate and copy the motorBike tutorial:

source /ichec/packages/OpenFOAM/OpenFOAM-X.Y.Z/etc/bashrc
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/motorBike ./
cd motorBike

where 'X.Y.Z' above should be replaced with the version number to be used. An example PBS script for this job is:

#!/bin/bash
#PBS -l nodes=1:ppn=24
#
#PBS -l walltime=00:30:00
#PBS -N myJobName
#PBS -A MyProjectName
#PBS -o outputFileName
#PBS -e errorFileName

module purge
module load apps
module avail
module load openfoam/X.Y.Z
source /ichec/packages/OpenFOAM/OpenFOAM-X.Y.Z/etc/bashrc

cd $PBS_O_WORKDIR

echo "Starting at "
date
if [ -d 0.org ]; then
cp -r 0.org/ 0
elif [ -d 0.orig ]; then
cp -r 0.orig/ 0
fi

# copy motorbike surface from resources folder
cp $FOAM_TUTORIALS/resources/geometry/motorBike.obj.gz constant/triSurface/

surfaceFeatureExtract >> surfaceFeatureExtract.log
blockMesh >> blockMesh.log
decomposePar >> decomposePar.log

mpiexec -np 6 snappyHexMesh -overwrite -parallel >> snappyHexMesh.log
ls -d processor* | xargs -i rm -rf ./{}/0 $1
if [ -d 0.org ]; then
ls -d processor* | xargs -i cp -r 0.org ./{}/0 $1
elif [ -d 0.orig ]; then
ls -d processor* | xargs -i cp -r 0.orig ./{}/0 $1
fi
mpiexec -np 6 patchSummary -parallel >> patch.log
mpiexec -np 6 potentialFoam -parallel -noFunctionObjects -writep >> potentialFoam.log
mpiexec -np 6 simpleFoam -parallel >> simpleFoam.log
reconstructParMesh -constant >> reconstructParMesh.log
reconstructPar -latestTime >> reconstructPar.log

echo "Run completed at "
date

This job can be submitted using the command

qsub scriptname.pbs

where scriptname.pbs is the filename of the sample PBS script

Setting Environment Variables

Frequent OpenFOAM users will know that the custom linux environment is set by sourcing the bashrc contained within the source code. However, to ensure full capability of OpenFOAM the user might also need to set the $WORK path before sourcing the bashrc. This is done with a command similar to

export WORK=/ichec/work/my_projectname/my_username

Post-Processing

On Fionn, OpenFOAM usually is not compiled with the included Paraview package due to performance considerations. We advise users to simply create the reader module using

paraFoam -touch

then transfer their case data to their local machine using a utility such as scp (possibly in combination with tar command as discussed in this link) or sshfs.


Additional Notes

Further information can be obtained from the OpenFOAM Foundation Website www.openfoam.org.

Return to the software index