The module system

When you execute a (non-builtin) command in bash, the location of the program corresponding to that command is looked up in the filesystem according to the $PATH environment variable.  For instance, on a typical Linux machine, one might see

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Thus, when the command 'minctracc' is typed into a Bash session, the above ':'-separated list of directories will be searched in left-to-right order for an executable file (script or binary) called 'minctracc'.

On most systems (NixOS/nixpkgs and Guix being the notable exceptions), software is simply upgraded in-place.  However, for scientific computing, we must be able to install multiple versions of the same software (together with their dependencies, of course).  For instance, users may want to test a new version or use features not available in previous versions, but must also be able to use old versions to continue and re-run previous analyses.  On the other hand, upgrading system software such as that in /bin and /sbin is necessary for security reasons.  We currently resolve these issues by providing a collection of environment modules (modules for short), although arguably use of lightweight containers (Singularity/Apptainer, Docker, Podman, etc.) or a Nixpkgs-like installation is a better solution.

Currently, (August 2022) we have Ubuntu 20.04 (focal) installed, and our science software available via modules resides here:

/projects/tools/programs/ubuntu-20.04/spack/share/spack/lmod/linux-ubuntu20.04-x86_64

Accessing the module system

In brief:

module load StdEnv

Here StdEnv (formerly called mice-env, and still called that on non-MICe systems such as HPF, Graham, etc., which may define their own StdEnv) is an omnibus module which loads a variety of other modules.

StdEnv is no longer loaded automatically, as used to be the case during mice-env days.  The main reason is that StdEnv provides many more modules in order to increase isolation from the "mutable" base OS packages, and is slower to load as a result.  If you are fine with every terminal taking several seconds to start, feel free to add 'module load StdEnv' to your ~/.bashrc – there are no other known issues involving this (although in principle some system libraries such as bz, lz4 etc might break as a few modules set LD_LIBRARY_PATH in order for Python cdll to find their contents).  We ought to investigate ways to optimize this load time. 

You can list available modules by using the following command:

module avail

Similarly, `module list` lists 'loaded' modules, 'module purge' unloads all modules.  You can also use 'ml' as a shorthand, e.g. 'ml mice-env', 'ml list', etc., but this doesn't work with older module systems such as at HPF.  For more details, see https://lmod.readthedocs.io/en/latest/.

By default, the most recent version of the StdEnv module is loaded.  When beginning a project, it is likely a good idea to run 'module avail' and note the version of this module in order to reload the same version for the remainder of the project in order to prevent, e.g., unexpected type errors in project-specific R or Python code or even command-line tools, changes in numerics affecting your results, etc.  You can create a bash script to reload this module and even use the direnv  program to automatically reload whenever you re-enter this project directory.

source /etc/profile.d/z00-lmod.sh
module use /projects/tools/programs/ubuntu-20.04/spack/share/spack/lmod/linux-ubuntu20.04-x86_64/Core/

Modules and Pydpiper

For information about registration quarantines and pydpiper versions look here: PydPiper at MICe with quarantines