GLIBC and the container-based workaround

The problem

Have you ever tried to install/use a software on Mistral and seen a message like this?

/lib64/libc.so.6: version `GLIBC_2.14` not found

This is for example one of the reasons why PyTorch is not available in our python3 module. Those software packages require a newer version of glibc. Unfortunately, most of Mistral nodes are based on CentOS 6 kernel. To check the version of glibc:

ldd --version

You will get an output like this:

ldd (GNU libc) 2.12
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

So, how can you in some cases overcome this issue?

The workaround

Singularity containers

For many of the issues with glibc, we propose the use of singularity containers. The advantage is that you can use different Operating Systems running a higher version of glibc per default. Example:

  • CentOS 7

  • Ubuntu 16

Prerequesites

  • image with pytorch installed (you can also create it and then push it to Mistral)

  • os with a kernel supported on Mistral

The PyTorch case

Let’s take the simplest way:

  1. you search for a container image with pytorch installed (Dockerhub, Singularityhub)

  2. check if it works on Mistral (Kernel)

  3. load singularity module on MIstral

  4. pull the image

  5. run/execute the container

  6. use your software

Step (1) can be more complicated depending on your workflow and which other softwares are required: gpu, cuda, cudnn, nvidia drivers.

Here a concrete example:

% module load singularity/3.6.1-gcc-9.1.0
% singularity pull --name pytorch_gpu.simg shub://yinglilu/pytorch_gpu_singularity:1.3.0
INFO:    Use cached image
% singularity shell pytorch_gpu.simg
Singularity> python
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda custom (64-bit) on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>>

If you check the version of glibc inside the container, you will notice it has changed to a higher version.

Singularity> ldd –version ldd (Debian GLIBC 2.24-11+deb9u4) 2.24
Copyright (C) 2016 Free Software Foundation, Inc. This is free software;
see the source for copying conditions. There is NO warranty; not even
for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by
Roland McGrath and Ulrich Drepper. Singularity> uname -r
2.6.32-754.29.2.el6.x86_64 Singularity> cat /etc/os-release
PRETTY_NAME=“Debian GNU/Linux 9 (stretch)NAME=“Debian GNU/Linux”
VERSION_ID=“9” VERSION=“9 (stretch)ID=debian
HOME_URL=“https://www.debian.org/”
SUPPORT_URL=“https://www.debian.org/support”
BUG_REPORT_URL=“https://bugs.

The Caffe framework case

Another kernel related issue occurred when trying to build the [Caffe deep learning framework] on Mistral. Hopefully, they have some Docker images which are supported on Mistral (after conversion to singularity).

Example with the cpu version:

% module load singularity/3.6.1-gcc-9.1.0
% singlularity pull docker://bvic/caffe:cpu
% singularity run caffe_cpu.sif

Final remarks

Note

“Assumption” In this article, I assume that you are already familiar with the concept of container images and virtualization.

Important

Please take a look here for more information on how to use singularity on Mistral.

Note

Currently, containers are seen as workaround for some issues on Mistral. So, do not expect they will solve all problems.