Anaconda and Singularity

Owned by Jeff Polasek
Last updated: May 09, 2023
2 min read

There are two tools for building an environment for using the GPU’s on Olympus, Anacona/Miniconda and Singularity.

Please install anaconda in your /mnt/shared-scratch directory. Anaconda environments can become large and fill up your home directory.

Anaconda/Miniconda

Anaconda is a distribution of the Python and R programming languages for scientific computing (data science, machine learning applications, large-scale data processing, predictive analytics, etc.), that aims to simplify package management and deployment.

Anaconda is a powerful tool that loads the correct versions of python modules to ensure compatibility in your environment. For example, you can create different virtual environments for different versions of python, tensor-flow or pytorch.

One limitation of Anaconda is there are a limited number of versions available for tensor-flow and pytorch.

Singularity

Singularity is a container system similar to Docker. Singularity containers allows users to run applications in a Linux environment of their choosing. Containers can run different different distros (Ubuntu, CentOS, Mint) on a single system. This provides more flexibility than Anaconda, but can be more difficult to configure as since there are no versioning tools to ensure compatibility.

Should I use Anaconda or Singularity?

In many cases, Anaconda (or Miniconda) is the best tool to configure your machine learning environment.

For beginning users, good reference for getting started with Anaconda is

https://conda.io/projects/conda/en/latest/user-guide/getting-started.html

The following site provides instructions for installing anaconda

https://astrobiomike.github.io/unix/conda-intro

However, there are cases where a singularity container is the better choice. Some of these cases include needing to use a different Linux distribution/version number, being unable to configure anaconda to match the versions needed for specialty python modules, or requiring specialty applications.

More information on using singularity containers on Olympus is located at

https://tamuengr.atlassian.net/wiki/spaces/helpdesk/pages/1987510273