Using Singularity
The latest release of SingularityCE v.3.9.8 maintained by Sylabs is installed on all SOMHPC nodes default to all users PATH.If the system Singularity version does not meet your lab's workflow, you can do a self-install in a user conda environment via conda-forge
Quick Test
cycle_admin@ip-0A260C0B:~$ singularity version
3.9.8-bionic
cycle_admin@ip-0A260C0B:~$ singularity run docker://godlovedc/lolcow
_______________________________________
/ You will be aided greatly by a person \
\ whom you thought to be unimportant. /
---------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
You should see a random generated quote from lolcow displaying in user terminal
Singularity CLI and Running Containers
Singularity command line interface (CLI) uses positional syntax (i.e. the order of commands and options matters). Global options follow the main singularity command. Then sub commands are followed by their options and arguments.
$ singularity [global options...] subcommand [subcommand options...] ...
Run ephemeral container with run command or run the image as if it were an executable
$ singularity run lolcow_latest.sif
______________________________
< Mon Aug 16 13:01:55 CDT 2021 >
------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
$ ./lolcow_latest.sif
______________________________
< Mon Aug 16 13:12:50 CDT 2021 >
------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Download image with pull or build. With build you need to specify a name for the container.
$ cycle_admin@ip-0A260C0B:~$ singularity pull docker://sylabsio/lolcow INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures Copying blob 16ec32c2132b done Copying blob 5ca731fc36c2 done Copying config fd0daa4d89 done Writing manifest to image destination Storing signatures 2022/04/21 00:13:04 info unpack layer: ... 2022/04/21 00:13:06 info unpack layer: ... INFO: Creating SIF file... cycle_admin@ip-0A260C0B:~$ singularity build lolcow docker://sylabsio/lolcow INFO: Starting build... Getting image source signatures Copying blob 5ca731fc36c2 skipped: already exists Copying blob 16ec32c2132b skipped: already exists Copying config fd0daa4d89 done Writing manifest to image destination Storing signatures 2022/04/21 00:45:14 info unpack layer: ... 2022/04/21 00:45:16 info unpack layer: ... INFO: Creating SIF file... INFO: Build complete: lolcow
Building a Container
To build a sandbox container, or a .sif image in Singularity requires sudo on the host system. For a normal user, having access to root on a shared cluster may not be possible. The requirement for sudo has been largely discussed in the community as seen in here and here.
cycle_admin@ip-0A260C0B:~/lolcow$ singularity build --sandbox lolcow lolcow.def FATAL: You must be the root user, however you can use --remote or --fakeroot to build from a Singularity recipe file
It is not uncommon users develop and build the container in their own machine where they have full access to their own environment, and only pull the image into HPC cluster ready to run.
Here we discuss 3 alternative options - all free - that a user can develop and build their container.
- use
--fakerootflag - use
--remoteflag - Sylabs.io builder
fakeroot flag
Fakeroot flag is introduced in Singularity v3.5+, read here for more information. This feature allows you to build and operate the container (interactively) as if you are root
For this feature to work:
You need to submit a request for your
netidto be added in/etc/subuidand/etc/subgidnamespace mappingStart an interactive bash session on a compute node to build the container (do not build on scheduler node)
--fakerootcannot be used on a shared filesystem (i.e./homeor/data) - create a personal folder under/tmpinstead
cycle_admin@ip-0A260C0B:/tmp/lolcow$ singularity build --fakeroot lolcow2 lolcow.def INFO: Starting build... ... ... INFO: Adding runscript INFO: Creating SIF file... INFO: Build complete: lolcow2 start a shell session in the container as root cycle_admin@ip-0A260C0B:/tmp/lolcow$ singularity shell --fakeroot --writable /tmp/lolcow/lolcow Singularity> id uid=0(root) gid=0(root) groups=0(root)
remote flag
With remote flag, your definition file is sent to Sylabs.io where the container is built on the cloud. You need to create an account on Sylabs.io and generate an access token to use this flag.
cycle_admin@ip-0A260C0B:/tmp/lolcow$ singularity build --remote lolcow2 lolcow.def FATAL: Unable to submit build job: no authentication token, log in with `singularity remote login` cycle_admin@ip-0A260C0B:/tmp/lolcow$ singularity remote login Generate an access token at https://cloud.sylabs.io/auth/tokens, and paste it here. Token entered will be hidden for security. Access Token: INFO: Access Token Verified! INFO: Token stored in /shared/home/cycle_admin/.singularity/remote.yaml Once logged in with an access token, you can proceed to build with --remote cycle_admin@ip-0A260C0B:/tmp/lolcow$ singularity build --remote lolcow2 lolcow.def INFO: Remote "cloud.sylabs.io" added. INFO: Access Token Verified! INFO: Token stored in /root/.singularity/remote.yaml INFO: Remote "cloud.sylabs.io" now in use. INFO: Starting build...
Sylabs.io builder
This option will take the buildng phase out of HPC cluster, but the tool is completely free to use on Sylabs.io website up to a fixed amount of storage
https://cloud.sylabs.io/builder
References
https://sylabs.io/guides/latest/user-guide/quick_start.html#overview-of-the-singularityce-interface
https://sylabs.io/guides/latest/user-guide/quick_start.html#running-a-container