BLCR (Berkeley Lab Checkpoint/Restart) is a Linux kernel module whose goal is to supply a robust production quality library to checkpoint a large variety of applications without needing to modify their source code. The module concentrates on checkpointing parallel applications that communicate using MPI.
- TCP/IP sockets are not saved or reopened.
- No support for SysV IPC calls (interprocess call)
- Zombie processes are not restored.
- No support for statically linked applications.
- No support for MPI-2 calls (one-sided communications and MPI-IO).
- A write operation is performed for each contiguous unit of virtual memory. If your application contains complex data structures (for example, linked lists or trees), checkpointing performance may be limited.
The script easy_blcr.sh (Colosse only)
To make it easier to perform checkpoints and restarts using BLCR, Colosse's team has written a script that makes this procedure very simple for the user. You only need to source the file /clumeq/bin/easy_blcr.sh and run your program using the commands ckpt_run or ckpt_run_task after having set certain parameters in your submission script. The principle is relatively simple. This command launches the application using BLCR, then runs a daemon who observes the application's execution and performs checkpoints at regular intervals. When the task runs, the script automatically detects the presence of old checkpoints and relaunches from the last one if it exists.
Here is an example where checkpointing is done automatically every hour.
The command ckpt_run runs a task in the background, and then runs a daemon, who performs checkpoints at regular intervals. The last line ensures that when the daemon runs, the script continues. Interacting with the daemon is possible by sending signals to the job.
Example #2, with multiple tasks
You have multiple serial tasks on one node? No problem. You can run them directly using the command ckpt_run_task, as follows:
Note that in this case, you should pass an additional parameter to ckpt_run_task, which indicates the task number. This identifier must be unique. Checkpoints will be created in a subdirectory that has the name of this identifier within the checkpoint directory. You should also not put the task in the background yourself (using the symbol & at the end of the line): the script does that automatically.
Interacting with the daemon: forcing a checkpoint
When you use the script /clumeq/bin/easy_blcr.sh, you can force a checkpoint for a running job using the command mjobctl -N SIGCONT jobid. This sends a signal to the script asking it to checkpoint. This also works well if you use ckpt_run or ckpt_run_task.
- The same limitations as BLCR.
- The script easy_blcr.sh does not support MPI jobs.