Migrating cross-platform conda environment

Owned by Jimmy Hung
Last updated: Apr 07, 2023 by Terri West
1 min read

When a user creates a conda environment with exported YML file from a different OS platform (Azure HPC OS is Ubuntu 20.04 and HARDAC OS is CentOS7), the exported conda build dependencies may cause conflicts with on the new system. Example below:

Encountered problems while solving.
Problem: package libblas-3.8.0-14_openblas has constraint blas * openblas conflicting with blas-1.0-mkl
Problem: package libzlib-1.2.11-h166bdaf_1014 has constraint zlib 1.2.11 *_1014 conflicting with zlib-1.2.11-ha838bed_2
Problem: package numpy-base-1.15.2-py37h81de0dd_1 requires blas 1.0 mkl, but none of the providers can be installed
Problem: package scipy-1.1.0-py37hfa4b5c9_1 requires blas 1.0 mkl, but none of the providers can be installed

Here we provide a system-denpendency-free conda environment migration to address the issue:

  1. Use --from-history to only export explicit dependencies. (This requires conda v4.7.12 or later)ca)t /etc/os-release

    $ conda env export --from-history > old_cluster_env.yml

    Use --prefix flag to export from a file path

    $ conda env export --from-history --prefix /data/envs/xyz_env > /data/exports/export_xyz.yml
  2. Copy the exported file over to Azure HPC cluster

  3. Initialize one-time bash script with conda

    $ /sched/anaconda3/bin/conda init bash

    Exit and re-enter a new bash session will land you in the base conda environment. Base conda environment is a read-only environment.

  4. Inspect the exported YAML file name and prefix to specify the name of the environment and the location for the environment. (Due to latency issue with many-small-file writes with HPC cache, we highly suggest users to create conda environment under /tmp directory on an execute node (not login nodes) for testing before they create a conda environment in /data or /home directory. )

  5. Use mamba from base environment to create a new conda environment from the exported file

    (base)$ mamba env create -f old_cluster_env.yml

    Depending the size of your yml file, a realistic conda creation in /data can take between 20 minutes up to 5 hours.

References: