BLCR

De Wiki de Calcul Québec
Aller à : Navigation, rechercher
Cette page est une traduction de la page BLCR et la traduction est complétée à 100 % et à jour.

Autres langues :anglais 100% • ‎français 100%

Sommaire

Description

BLCR (Berkeley Lab Checkpoint/Restart) is a Linux kernel module whose goal is to supply a robust production quality library to checkpoint a large variety of applications without needing to modify their source code. The module concentrates on checkpointing parallel applications that communicate using MPI.

Limitations

  1. TCP/IP sockets are not saved or reopened.
  2. No support for SysV IPC calls (interprocess call)
  3. Zombie processes are not restored.
  4. No support for statically linked applications.
  5. No support for MPI-2 calls (one-sided communications and MPI-IO).
  6. A write operation is performed for each contiguous unit of virtual memory. If your application contains complex data structures (for example, linked lists or trees), checkpointing performance may be limited.

The script easy_blcr.sh (Colosse only)

To make it easier to perform checkpoints and restarts using BLCR, Colosse's team has written a script that makes this procedure very simple for the user. You only need to source the file /clumeq/bin/easy_blcr.sh and run your program using the commands ckpt_run or ckpt_run_task after having set certain parameters in your submission script. The principle is relatively simple. This command launches the application using BLCR, then runs a daemon who observes the application's execution and performs checkpoints at regular intervals. When the task runs, the script automatically detects the presence of old checkpoints and relaunches from the last one if it exists.

Example #1

Here is an example where checkpointing is done automatically every hour.


File : submit-blcr.sh
#!/bin/bash
#PBS -S /bin/bash
#PBS -N NAME      # Job name
#PBS -A xxx-yyy-zz   # RAP ID
#PBS -l walltime=24:00:00    # Duration in seconds
#PBS -l nodes=1:ppn=8  # Number of processors. Must be 8.
 
cd $PBS_O_WORKDIR
 
 
source /clumeq/bin/easy_blcr.sh
####################
## CONFIGURE
####################
CKPT_DELAY=60 					# Delay between checkpoints, in minutes
#CKPT_KEEP_BACKUPS=1 				# Do we keep backups of all checkpoints ?
#CKPT_DIR=checkpoints 				# Directory for the checkpoints
#CKPT_FILE=ckpt 				# Filename for the checkpoints
#CKPT_RESTART_FILE=$CKPT_FILE  		  	# Restart from this checkpoint file
#CKPT_LOG_FILE=ckpt.log 			# Log file
####################
 
ckpt_run ./my_executable param1 param2
 
ckpt_wait


The command ckpt_run runs a task in the background, and then runs a daemon, who performs checkpoints at regular intervals. The last line ensures that when the daemon runs, the script continues. Interacting with the daemon is possible by sending signals to the job.

Example #2, with multiple tasks

You have multiple serial tasks on one node? No problem. You can run them directly using the command ckpt_run_task, as follows:

File : submit-blcr.sh
#!/bin/bash
#PBS -S /bin/bash
#PBS -N NAME      # Job name
#PBS -A xxx-yyy-zz   # RAP ID
#PBS -l walltime=24:00:00    # Duration in seconds
#PBS -l nodes=1:ppn=8  # Number of processors. Must be 8.
 
cd $PBS_O_WORKDIR
 
source /clumeq/bin/easy_blcr.sh
####################
## CONFIGURE
####################
CKPT_DELAY=60 					# Delay between checkpoints, in minutes
#CKPT_KEEP_BACKUPS=1 				# Do we keep backups of all checkpoints ?
#CKPT_DIR=checkpoints 				# Directory for the checkpoints
#CKPT_FILE=ckpt 				# Filename for the checkpoints
#CKPT_RESTART_FILE=$CKPT_FILE  		  	# Restart from this checkpoint file
#CKPT_LOG_FILE=ckpt.log 			# Log file
####################
 
ckpt_run_task 1 ./my_executable param1
ckpt_run_task 2 ./my_executable param2
ckpt_run_task 3 ./my_executable param3
ckpt_run_task 4 ./my_executable param4
ckpt_run_task 5 ./my_executable param5
ckpt_run_task 6 ./my_executable param6
ckpt_run_task 7 ./my_executable param7
ckpt_run_task 8 ./my_executable param8
 
ckpt_wait


Note that in this case, you should pass an additional parameter to ckpt_run_task, which indicates the task number. This identifier must be unique. Checkpoints will be created in a subdirectory that has the name of this identifier within the checkpoint directory. You should also not put the task in the background yourself (using the symbol & at the end of the line): the script does that automatically.

Interacting with the daemon: forcing a checkpoint

When you use the script /clumeq/bin/easy_blcr.sh, you can force a checkpoint for a running job using the command mjobctl -N SIGCONT jobid. This sends a signal to the script asking it to checkpoint. This also works well if you use ckpt_run or ckpt_run_task.

Limitations

  1. The same limitations as BLCR.
  2. The script easy_blcr.sh does not support MPI jobs.

Useful links

Outils personnels
Espaces de noms

Variantes
Actions
Navigation
Ressources de Calcul Québec
Outils
Partager