Note: This documentation has been tested on Colosse. Certain instructions could be different on other servers.
OpenFOAM (Open Field Operation and Manipulation) is a multi-physics code that is largely focused on solving the equations of fluid mechanics. It's an MPI application which is well-adapted to running in parallel on a cluster.
It's an open source code.
OpenFOAM a some peculiarities that should be understood to use the software effectively on a cluster and not disrupt the stability of the cluster.
Creation of Many Files
OpenFOAM generates a great many output files. Creating a checkpoint leads to the creation of a directory for each MPI process containing a file per parameter. When running on 64 or 128 cores, you can create therefore several hundred files at each checkpoint, which has the potential to result in thousands or even tens of thousands of files after just a few hours of execution. While these files are small in size they can saturate the filesystem and cause you to exceed your disk quota on a server.
Creation of an Important Load on a Parallel Filesystem
It's important to realize that each operation on a file generates one or more input/output (I/O) operations on the underlying filesystem. Examples of I/O operations include: opening or closing a file (open, close), writing or reading the contents of a file (read, write) and inquiring about the properties of a file (stat). A given filesystem can only respond to a certain number of I/O operations in a given time interval. This value can be measured in terms of I/O operations per second (iops). An application which creates a large number of iops leads to a significant load on the filesystem and will thus have a negative effect on its own performance as well as on the performance of the filesystem for other users.
By design, OpenFOAM generates an elevated number of iops. We have observed loads, depending on the problem being solved and the parameters used, ranging from 50 to 50000 iops per compute node used (eight cores). The effect of this on other users can be considerable.
Some advice to obtain the best possible performance from OpenFOAM and minimize any negative effects on the server.
- Reduce the frequency of checkpointing.
- Reduce the number of checkpoint files that are saved during the execution of a job: it's possible to significantly reduce the number of checkpoints by means of the parameter purgeWrite in the file system/controlDict. This parameter allows you to control the number of checkpoints that are kept during the computation. When purgeWrite is equal to zero, all of the checkpoints are kept. However, when the parameter has a value greater than zero, this positive integer indicates the number of checkpoints to keep. Older checkpoints are deleted when more recent ones are written. This allows you to have a high frequency of checkpoints without creating an excessive number of files.
- Rapidly carry out post-processing on the results of your computations: it's a good idea to rapidly do the post-processing of your OpenFOAM results so that you can aggregate the results into fewer files. This will let you rapidly reduce the number of files stored on the system and therefore to continue to submit other jobs without exceeding the limit on the number of files you can have.
- Avoid having a very small time step: the duration in CPU time of a time step in your simulation will have an important effect on the performance of your job. At the end of each time step the data are synchronized and a log file entry is written, so that reducing the number of time steps will reduce the proportion of wall time that is spent on such bookkeeping tasks compared to real calculations. It's beneficial therefore to aim for the largest possible value of the time step, which can be done for example by increasing the number of cells per core.
- Using OpenFOAM without permitting on-the-fly modifications: by default, OpenFOAM lets you modifier the problem parameters on the fly, i.e. during execution. Using this functionality however obliges OpenFOAM to watch the input files throughout its execution and therefore to inquire about the properties (stats) at each time step. When this is combined with a very brief time step, it can lead to tens of thousands of iops for no purpose. Though very useful when developing and testing a model for the exact parameters of a desired simulation, it's preferable to disable this functionality to improve the performance of OpenFOAM. Tests carried out locally have shown that disabling this on-the-fly modification functionality resulted in reducing by a factor of 10 the number of iops created by OpenFOAM as well as a 20% reduction in the calculation time for a given problem. The option runTimeModifiable in the file system/controlDict allows you to control this functionality.
Thanks to Professor Guy Dumas and his research group at the Department of Mechanical Engineering of Laval University for the creation of these recommendations and especially Simon Lapointe.
OpenFOAM 2.1.0 Module
Certain environment variables aren't defined by this module out of a desire to keep the user environment as coherent as possible. These variables normally point towards various site and user directories. Here is a list of these environment variables as well as their standard value. The use of these variables isn't necessary if you don't have any additional content.
To add these variables to your environment, use the export command. For example, to add WM_PROJECT_USER_DIR,
[name@server $] export WM_PROJECT_USER_DIR=$HOME/$WM_PROJECT/$USER-$WM_PROJECT_VERSION
Note that the right hand side of the above equation can take any value.
If you add FOAM_SITE_APPBIN, FOAM_SITE_LIBBIN, FOAM_USER_APPBIN and FOAM_USER_LIBBIN, don't forget to add them to your PATH and LD_LIBRARY_PATH.
One alias isn't defined either, run. To define it,
[name@server $] alias run "cd $FOAM_RUN"
It is always possible (but not necessary) to source the default configuration by using the environment variable OPENFOAM_ETC like this,
[name@server $] source $OPENFOAM_ETC/bashrc