The Loadleveler Job Management System

Parallel programs can not be run interactively on the Blue Gene/P "genius". All jobs on the Blue Gene/P have to be submitted to the LoadLeveler batch system. The LoadLeveler will allocate the resources required for your job (e.g. the compute nodes to run your job on).

By default, the job run limit is set to 1 on "genius". If your batch jobs can run independently from each other, your job run limit can be raised on request.

To run a job on the Blue Gene/P the following LoadLeveler keywords are required in your job script:

1. You have to indicate your job as a BlueGene job with

   # @job_type=bluegene

   (Otherwise the job is executed as a serial job on the login node genius.rzg.mpg.de without allocating a BlueGene partition.)


2. The size of a job (i.e. the number of compute nodes the job will use) has to be specified by using

   # @bg_size=value

   The following values are allowed: 128 (512 cores), 256 (1024 cores), 512 (2048 cores), 1024 (4096 cores), 2048 (8192 cores)


3. The topology of the partition can be specified with the keyword

   # @bg_connection=value

   which can be one of the three values: MESH, TORUS or PREFER_TORUS.
   A TORUS partition can lead to a better performance of your application than a MESH partition. In case of doubt always add

   # @bg_connection=PREFER_TORUS

   to your job script.

The most important Loadleveler commands are:

llsubmit

Submit a job script for execution. See for exampe the sample script.

llq

llq -b

Check the status of your BlueGene job(s).

llcancel

Cancel a job.

llclass

This command provides a list of the available batch classes. (This list is likely to be more up-to-date than our web documentation.)

There is also a graphical user interface xloadl (under X11) that you may find more convenient to use.

On the Blue Gene/P, parallel programs have to be started with mpirun. Hints:

• since the LoadLeveler automatically selects the appropriate partition for the job, the "–partition" option must not be specified in the mpirun command. 
• environment variables need to passed to the job using mpirun -env VARIABLE=VALUE .... A simple export VARIABLE=VALUE in the Loadleveller batch script is not sufficient!
  
• See mpirun -h for more information on the mpirun command.

Parallel codes can use the Blue Gene compute nodes in 3 different modes:
   SMP mode: one MPI process per node, 2048 MB memory, up to 4 threads. This is the default.
   DUAL mode: two MPI processes per node, 1024 MB memory per process, up to two threads.
   VN (Virtual Node) mode: four MPI processes per node, 512 MB per process; no additional threads.

   Example: mpirun -mode VN -exe ./myprog

Sample Batch job script : sample script.