Using Singularity for Container Support

Singularity vs. Docker at a Glance

Singularity takes a fundamentally different approach to security and permissions compared to Docker. Most notably, Singularity preserves a user’s rights and permissions whenever it enters a container. This means that a user inside of a container is subject to the same restrictions they are outside of the container; namely, if a user does not have root permissions outside of the container, they have no way of gaining them inside. This approach to security is what allows us to deploy Singularity to managed computers without leaving security holes. 

One of Singularity’s more interesting features is its ability to make use of both its own container format, as well as make use of Docker containers by converting them to the Singularity format when they are pulled from the Docker Hub. This allows you to make use of Docker containers even without having Docker installed on your computer.

It is very important to keep in mind when using Singularity, especially when using a converted Docker container, that if you would not have the permissions required to run a command or access a resource outside of a container, you will not be able to do so inside of the container. 

Building a Container

Singularity Containers (.sif)

Containers from the Library Hub

Pre-built container images can be found on the Sylabs Library Hub. You can download container files easily into your current directory by running:

Containers from a Definition File

Containers can be built from a container definition file, which specifies a base for the container to be built on and any software to install and environment variables to set. More information about Container Definitions can be found here: Definition Files — Singularity container 3.0 documentation  

Docker Containers

Singularity also supports using Docker containers. It can convert Docker images from the Docker Hub to Singularity’s container format.

The container will be saved as “<desired name>.sif” in the current directory. 

Using a Container

Once a container is built, it can be run in several ways. The container can be run persistently, or you can open an interactive shell in the container. By default, Singularity mounts the user’s home directory and a few system directories as read-write; all non-mounted directories are read-only by default. Please note that, unlike Docker, Singularity maintains user permission inside of containers, which means you can only read and write to directories that you have the necessary permissions.

Interactive Shell (shell)

You can open an interactive shell inside of a container image by executing:

Persistent Container (instance)

Persistent containers are beneficial if you have tasks which run continuously in the background, or if they are set up to require no interaction from the user after being started. It is important to note that instances can only be accessed and stopped by the user that started them. You can create an instance of any container by running:

Once an instance has been started, if any commands were specified to execute upon starting an instance, they will execute. Otherwise, you can start a shell to access the instance:

You can view all running instances you have started by executing: 

You can stop a single instance by executing:

Or you stop all running instances by executing:

Single Command Inside Container (exec)

To execute a single command inside of a container without creating an instance or starting an interactive shell, execute:

Write Support Inside Container

By default, containers are read-only outside of any directories that are mounted when accessing the container. To enable writing to the container, include the --writable flag when executing instance start, shell, or exec. 

If you don’t need to preserve files after leaving the container, you can instead use the --writable-tmpfs flag. This allows writing to the container directories, but only temporarily. These files will be deleted when the container closes. 

Binding Directories to a Container

By default, Singularity mounts the user’s home directory and a few system directories into the container. These are mounted as read-write, however normal user permissions still apply. If you need to mount additional directories into the container, this can be accomplished using the --bind flag. This flag can be used with the instance start, shell, and exec subcommands. An example:

More information can be found at Bind Paths and Mounts — Singularity container 3.3 documentation  

Isolating a Container

To prevent Singularity from mounting the home directory when entering the container, you can use the --containall flag with any of the instance start, shell, or exec subcommands. 

Python Virtual Environments in Containers

All machines managed by the Linux Team have support for creating Python virtual environments by default. Not all containers include the packages necessary to create them, so make sure you create a virtual environment first before starting your container. 

Create the virtual environment by running: 

You will want to upgrade the version of pip in the virtual environment first before using it in your container in order to make sure you’re obtaining the most recent versions of Python packages. Run the following commands:

Once you have created the virtual environment, start your container.

After you have entered the container, you can then enter the Python virtual environment:

Note for Containers Which Include Python Packages

If your container already includes Python packages which you will want to use (such as with various machine learning containers), you will want to enable access to the global package directory in your virtual environment. When you create your virtual environment, instead use the following command:

More Information on Python Virtual Environments

If you would like more information about using Python virtual environments, please see the document at Virtual Environments in Python.

NVIDIA CUDA Docker Support

NVIDIA provides a collection of Docker images that can be used for CUDA applications on computers with NVIDIA GPUs. Because of Singularity’s ability to convert Docker images to Singularity’s format, you can make use of these containers. 

Obtaining Containers from Nvidia

1. The full listing of containers available from the Nvidia GPU Cloud can be found at Data Science, Machine Learning, AI, HPC Containers | NVIDIA NGC .

2. Once you have identified the container, you can download it by opening the container’s page and copying the pull tag. At the top of the page, there will be a box that reads “docker pull <pull tag>”.

3. Download the container by running: 

An example of the pull tag would be: nvcr.io/nvidia/pytorch:20.03-py3.

1. If you get the error “Invalid status code returned when fetching blob 401" when trying to run singularity pull, you will need to create an account for Nvidia GPU Cloud and set up an API key in order to download this container. Follow the instructions in the next section.

Setting Up an Nvidia GPU Cloud Account

1. Following the Getting Started guide on NVIDIA NGC NGC Getting Started.

2. Add the following to the bottom of the file ~/.bashrc:

Make sure you replace <NVIDIA NGC API key> with the API key you generated as part of step 1

1. Log out and log back into your computer. 

2. You should now be able to download the Nvidia containers.

More information about using NVIDIA’s Docker images with Singularity can be found at https://docs.nvidia.com/ngc/ngc-user-guide/singularity.html#singularity  

Using Python Virtual Environments with Nvidia Containers

Many of the machine learning containers from Nvidia already contain Python packages for the machine learning libraries. When Nvidia builds the containers, these packages are typically compiled from the library’s source code against the latest version of CUDA, so they are not always the same as what is available with pip. If you’re going to use a virtual environment, please read the Python Virtual Environments in Containers section of this document, and follow the instructions in Note for Containers Which Include Python Packages when creating the virtual environment.

Troubleshooting

Cannot Build Container with --fakeroot Option

If you receive errors about permissions when trying to build a container with the --fakeroot option, please contact the Linux Team for assistance with fixing the relevant files to allow the --fakeroot option to work on your machine.

OpenCV with Python

It’s possible you may encounter the error “ImportError: libSM.so.6: cannot open shared object file: No such file or directory" when trying to import the cv2 module for OpenCV in Python. To get around this, you can remove the "opencv-python" package with pip and install "opencv-python-headless" in its place. 

Missing Python Libraries with Virtual Environment in a Container

If you’re not able to import Python libraries using a virtual environment in a Singularity container, make sure you’ve created your virtual environment like the example in Note for Containers Which Include Python Packages.