Singularity

Versions Installed

Kay: 2.6.0

Description

Singularity enable users to use containers on HPC systems such as Kay. Using singularity you can make complicated software stacks portable. For more information about singularity refer here

Workflow

Singularity provide multiple workflows for building, deploying and using containers. In the following text, we have presented a generic workflow suitable to most users.

In this workflow you should build singularity images on your local computer (where you have sudo rights) and then copy the singularity images to Kay to run your simulations. To install singularity on your local computer refer here

Some examples are given below

 

Examples

1. Serial code

On Local Computer

Create a singularity definition file (e.g. serial.def) with the contents

BootStrap: docker
From: centos:7

%environment
  LC_ALL="en_US.UTF-8"
  LC_CTYPE="en_US.UTF-8"
  LANGUAGE="en_US.UTF-8"
  export LC_ALL LC_CTYPE LANGUAGE

%post
  yum -y update
  yum -y install epel-release
  yum repolist
  yum -y install cowsay

 

Create the singularity image (e.g. serial.simg) from the singularity definition file (serial.def), using the command

sudo singularity build serial.simg serial.def

 

Copy the singularity image (serial.simg) to Kay (e.g. home directory)

 

On Kay

In the same directory as the singularity image (e.g. home directory), create a Slurm submission file (e.g. submit_serial.sh) with the contents

#!/bin/sh

#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --account=MY_PROJECT_ID
#SBATCH --partition=DevQ
#SBATCH --job-name=serial
#SBATCH --output=log_slurm.txt

module load singularity/2.6.0

cd $SLURM_SUBMIT_DIR
singularity exec serial.simg cowsay 'Hello!'

Make sure to replace the MY_PROJECT_ID with your project ID in the slurm submission file (submit_serial.sh)


Submit a job, using the command

sbatch submit_serial.sh


Once the job finishes, the output (in the log_slurm.txt) will look like

 ________
< Hello! >
 --------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

 

 

2. MPI code within a single node (using container MPI)

On Local Computer

Create a C file (e.g. hello_world.c) with the contents

#include <mpi.h>
#include <stdio.h>
#include <utmpx.h>

int main(int argc, char** argv) {
  // Initialize the MPI environment
  MPI_Init(NULL, NULL);

  // Get the number of processes
  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);

  // Get the rank of the process
  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

  // Get the name of the processor
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int name_len;
  MPI_Get_processor_name(processor_name, &name_len);

  // Get the id of the core
  int core_id;
  core_id = sched_getcpu();

  // Print off a hello world message
  printf("host=%s, size=%d, rank=%d, core=%d\n", processor_name, world_size, world_rank, core_id);

  // Finalize the MPI environment.
  MPI_Finalize();
}

 

Create a singularity definition file (e.g. mpi_container.def) with the contents

BootStrap: docker
From: centos:7

%post
  yum -y update
  yum -y install openmpi3-devel
  ln -s /usr/lib64/openmpi3/bin/mpicc /usr/bin/mpicc
  ln -s /usr/lib64/openmpi3/bin/mpirun /usr/bin/mpirun

%files
  hello_world.c /var/tmp/hello_world.c

%post
  mpicc -o /usr/local/bin/hello_world /var/tmp/hello_world.c


Create the singularity image (e.g. mpi_container.simg) from the singularity definition file (mpi_container.def), using the command

sudo singularity build mpi_container.simg mpi_container.def

 

Copy the singularity image (mpi_container.simg) to Kay (e.g. home directory)

 

On Kay

In the same directory as the singularity image (e.g. home directory), create a Slurm submission file (e.g. submit_mpi_container.sh) with the contents

#!/bin/sh

#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --account=MY_PROJECT_ID
#SBATCH --partition=DevQ
#SBATCH --job-name=mpi_container
#SBATCH --output=log_slurm.txt

module load singularity/2.6.0

cd $SLURM_SUBMIT_DIR
singularity exec mpi_container.simg /usr/bin/mpirun -np 40 /usr/local/bin/hello_world

Make sure to replace the MY_PROJECT_ID with your project ID in the slurm submission file (submit_mpi_container.sh)


Submit a job, using the command

sbatch submit_mpi_container.sh


Once the job finishes, the output (in the log_slurm.txt) will look like

host=n1, size=40, rank=10, core=2
host=n1, size=40, rank=16, core=12
host=n1, size=40, rank=24, core=14
host=n1, size=40, rank=37, core=33
host=n1, size=40, rank=38, core=11
host=n1, size=40, rank=5, core=28
host=n1, size=40, rank=6, core=17
host=n1, size=40, rank=17, core=22
host=n1, size=40, rank=20, core=15
host=n1, size=40, rank=26, core=13
host=n1, size=40, rank=27, core=34
host=n1, size=40, rank=30, core=0
host=n1, size=40, rank=36, core=3
host=n1, size=40, rank=39, core=30
host=n1, size=40, rank=0, core=7
host=n1, size=40, rank=1, core=26
host=n1, size=40, rank=2, core=19
host=n1, size=40, rank=3, core=36
host=n1, size=40, rank=4, core=8
host=n1, size=40, rank=7, core=35
host=n1, size=40, rank=8, core=18
host=n1, size=40, rank=9, core=20
host=n1, size=40, rank=12, core=16
host=n1, size=40, rank=13, core=32
host=n1, size=40, rank=14, core=5
host=n1, size=40, rank=15, core=24
host=n1, size=40, rank=18, core=8
host=n1, size=40, rank=19, core=24
host=n1, size=40, rank=21, core=23
host=n1, size=40, rank=22, core=10
host=n1, size=40, rank=23, core=37
host=n1, size=40, rank=25, core=38
host=n1, size=40, rank=28, core=17
host=n1, size=40, rank=29, core=29
host=n1, size=40, rank=31, core=21
host=n1, size=40, rank=32, core=4
host=n1, size=40, rank=33, core=36
host=n1, size=40, rank=34, core=1
host=n1, size=40, rank=35, core=20
host=n1, size=40, rank=11, core=22

 

 

3. MPI code over multiple nodes (using host and container MPI)

To run singularity containers with MPI over multiple nodes, the host MPI and the container MPI should be compatible. To simplify the preparation of singularity containers, you can use HPCCM on your local computer. For more information about HPCCM refer here.

On Local Computer

Create a C file (e.g. hello_world.c) with the contents

#include <mpi.h>
#include <stdio.h>
#include <utmpx.h>

int main(int argc, char** argv) {
  // Initialize the MPI environment
  MPI_Init(NULL, NULL);

  // Get the number of processes
  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);

  // Get the rank of the process
  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

  // Get the name of the processor
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int name_len;
  MPI_Get_processor_name(processor_name, &name_len);

  // Get the id of the core
  int core_id;
  core_id = sched_getcpu();

  // Print off a hello world message
  printf("host=%s, size=%d, rank=%d, core=%d\n", processor_name, world_size, world_rank, core_id);

  // Finalize the MPI environment.
  MPI_Finalize();
}


Create a HPCCM recipe file (e.g. mpi_host_container.py) with the contents

"""
MPI Bandwidth
Contents:
  CentOS 7
  GNU compilers (upstream)
  Mellanox OFED
  OpenMPI version 3.1.2
"""

Stage0 += comment(__doc__, reformat=False)

# CentOS base image
Stage0 += baseimage(image='centos:7')

# GNU compilers
Stage0 += gnu()

# Mellanox OFED
Stage0 += mlnx_ofed()

# OpenMPI
Stage0 += openmpi(configure_opts=['--with-slurm'], infiniband=False, cuda=False, version='3.1.2')

# MPI Hello World
Stage0 += copy(src='hello_world.c', dest='/var/tmp/hello_world.c')
Stage0 += shell(commands=['mpicc -o /usr/local/bin/hello_world /var/tmp/hello_world.c'])


Create the singularity definition file (e.g. mpi_host_container.def) from the HPCCM recipe file (mpi_host_container.py)

hpccm --format singularity --recipe ./mpi_host_container.py --singularity-version=2.6 > mpi_host_container.def

 

Create the singularity image (e.g. mpi_host_container.simg) from the singularity definition file (mpi_host_container.def), using the command

sudo singularity build mpi_host_container.simg mpi_host_container.def

 

Copy the singularity image (mpi_host_container.simg) to Kay (e.g. home directory)

 

On Kay

In the same directory as the singularity image (e.g. home directory), create a Slurm submission file (e.g. submit_mpi_host_container.sh) with the contents

#!/bin/sh

#SBATCH --time=01:00:00
#SBATCH --nodes=2
#SBATCH --account=MY_PROJECT_ID
#SBATCH --partition=DevQ
#SBATCH --job-name=mpi_host_container
#SBATCH --output=log_slurm.txt

module load singularity/2.6.0
module load openmpi/gcc/3.1.2

cd $SLURM_SUBMIT_DIR

mpirun -np 80 singularity exec mpi_host_container.simg /usr/local/bin/hello_world

Make sure to replace the MY_PROJECT_ID with your project ID in the slurm submission file (submit_mpi_host_container.sh)

 

Submit a job, using the command

sbatch submit_mpi_host_container.sh


Once the job finishes, the output (in the log_slurm.txt) will look like

host=n1, size=80, rank=6, core=5
host=n1, size=80, rank=27, core=25
host=n1, size=80, rank=8, core=0
host=n1, size=80, rank=19, core=21
host=n1, size=80, rank=24, core=11
host=n1, size=80, rank=37, core=26
host=n2, size=80, rank=66, core=2
host=n2, size=80, rank=45, core=31
host=n2, size=80, rank=47, core=28
host=n2, size=80, rank=62, core=15
host=n2, size=80, rank=64, core=5
host=n2, size=80, rank=69, core=38
host=n2, size=80, rank=77, core=30
host=n2, size=80, rank=79, core=34
host=n2, size=80, rank=44, core=16
host=n2, size=80, rank=50, core=7
host=n2, size=80, rank=53, core=27
host=n2, size=80, rank=57, core=20
host=n2, size=80, rank=59, core=39
host=n2, size=80, rank=60, core=4
host=n2, size=80, rank=71, core=37
host=n2, size=80, rank=78, core=3
host=n2, size=80, rank=48, core=6
host=n2, size=80, rank=49, core=36
host=n2, size=80, rank=51, core=35
host=n2, size=80, rank=52, core=9
host=n2, size=80, rank=55, core=31
host=n2, size=80, rank=67, core=23
host=n2, size=80, rank=73, core=33
host=n2, size=80, rank=40, core=17
host=n2, size=80, rank=42, core=15
host=n2, size=80, rank=54, core=8
host=n2, size=80, rank=56, core=0
host=n2, size=80, rank=61, core=21
host=n2, size=80, rank=63, core=38
host=n2, size=80, rank=72, core=10
host=n2, size=80, rank=74, core=12
host=n2, size=80, rank=76, core=14
host=n2, size=80, rank=41, core=26
host=n2, size=80, rank=43, core=27
host=n2, size=80, rank=46, core=9
host=n2, size=80, rank=58, core=11
host=n2, size=80, rank=65, core=28
host=n2, size=80, rank=68, core=0
host=n2, size=80, rank=70, core=19
host=n2, size=80, rank=75, core=25
host=n1, size=80, rank=0, core=10
host=n1, size=80, rank=1, core=35
host=n1, size=80, rank=2, core=12
host=n1, size=80, rank=3, core=39
host=n1, size=80, rank=4, core=1
host=n1, size=80, rank=5, core=33
host=n1, size=80, rank=7, core=37
host=n1, size=80, rank=9, core=29
host=n1, size=80, rank=10, core=15
host=n1, size=80, rank=11, core=31
host=n1, size=80, rank=12, core=19
host=n1, size=80, rank=13, core=32
host=n1, size=80, rank=14, core=16
host=n1, size=80, rank=15, core=36
host=n1, size=80, rank=16, core=6
host=n1, size=80, rank=17, core=24
host=n1, size=80, rank=18, core=4
host=n1, size=80, rank=20, core=3
host=n1, size=80, rank=21, core=23
host=n1, size=80, rank=22, core=17
host=n1, size=80, rank=23, core=22
host=n1, size=80, rank=25, core=27
host=n1, size=80, rank=26, core=11
host=n1, size=80, rank=28, core=8
host=n1, size=80, rank=29, core=39
host=n1, size=80, rank=30, core=9
host=n1, size=80, rank=31, core=33
host=n1, size=80, rank=32, core=5
host=n1, size=80, rank=33, core=38
host=n1, size=80, rank=34, core=13
host=n1, size=80, rank=35, core=38
host=n1, size=80, rank=36, core=2
host=n1, size=80, rank=38, core=18
host=n1, size=80, rank=39, core=25

 

License

Singularity is released under a standard 3 clause BSD license.

Benchmarks

N/A.

Additional Notes

To use singularity load the relevant environment module:

module load singularity/2.6.0

Further information can be found here.