How to install conda packages in the aarch64 from x86-64 architecture

Question

May I know how can I get the installed packages from a particular environment from x86-64 architecture linux and

how can I create a new conda environment in aarch64 architecture using the same package?

First in the x86-64 architecture linux machine called L2, I export the package

conda list --export > envconda.txt

When I open the envconda.txt, it is

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
_libgcc_mutex=0.1=main
_r-mutex=1.0.0=anacondar_1

I changed the platform : linux-64 to linux-aarch64 because I am going to install the packages in the aarch64 architecture.

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-aarch64
_libgcc_mutex=0.1=main
_r-mutex=1.0.0=anacondar_1

In aarch64 linux machine called L1, I create a conda environment conda create -n envtest --file envconda.txt

Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - setuptools==36.4.0=py36_1
  - kiwisolver==1.1.0=pypi_0
  - pyyaml==3.13=pypi_0
  - jedi==0.10.2=py36_2
  - libgcc==5.2.0=0
  - jsonschema==2.6.0=py36_0
  - ptyprocess==0.5.2=py36_0
  - prompt_toolkit==1.0.15=py36_0
  - libstdcxx-ng==9.1.0=hdf63c60_0
  - tqdm==4.36.1=pypi_0
  - tomli==1.2.3=pypi_0
  - astor==0.7.1=pypi_0
  - argparse==1.4.0=pypi_0
  - pycparser==2.19=pypi_0
  - testpath==0.3.1=py36_0
  - cudnn==7.6.5=cuda10.2_0
  - asn1crypto==0.22.0=py36_0
  - dataclasses==0.8=pypi_0
  - platformdirs==2.4.0=pypi_0
  - krbcontext==0.10=pypi_07
  - decorator==4.1.2=py36_0
  - lazy-object-proxy==1.7.1=pypi_0
  - gsl==2.2.1=0
  - pexpect==4.2.1=py36_0
  - icu==54.1=0
  - freetype==2.5.5=2
  - bleach==1.5.0=py36_0
  - matplotlib==3.1.1=pypi_0
  - wheel==0.29.0=py36_0
  - cudatoolkit==10.2.89=hfd86e86_1
  - glib==2.50.2=1
  - kneed==0.7.0=pypi_0
  - sqlite==3.13.0=0
  - importlib-metadata==1.7.0=pypi_0
  - python==3.6.2=0
  - jpeg==9b=0
  - pango==1.40.3=1
  - fontconfig==2.12.1=3
  - resampy==0.2.2=pypi_0
  - nbformat==4.4.0=py36_0
  - pixman==0.34.0=0
  - scikit-learn==0.21.3=pypi_0
  - termcolor==1.1.0=pypi_0
  - typed-ast==1.5.4=pypi_0
  - keras-applications==1.0.8=pypi_0
  - harfbuzz==0.9.39=2
  - libffi==3.2.1=1
  - jupyter_client==5.1.0=py36_0
  - gssapi==1.6.9=pypi_0
  - curl==7.54.1=0
  - keras==2.2.4=pypi_0
  - isort==5.10.1=pypi_0
  - simplegeneric==0.8.1=py36_1
  - joblib==0.14.0=pypi_0
  - pypandoc==1.6.3=pypi_0
  - python-dateutil==2.8.2=pypi_0
  - ipython_genutils==0.2.0=py36_0
  - pyparsing==2.4.2=pypi_0
  - ca-certificates==2022.6.15=ha878542_0
  - krb5==1.13.2=0
  - path.py==10.3.1=py36_0
  - markdown==3.0.1=pypi_0
  - requests-kerberos==0.12.0=pypi_0
  - hdfs==2.5.8=pypi_0
  - traitlets==4.3.2=py36_0
  - tornado==4.5.2=py36_0
  - librosa==0.7.0=pypi_0
  - pyasn1==0.4.8=pypi_0
  - blas==1.0=mkl
  - zlib==1.2.11=0
  - libogg==1.3.2=h14c3975_1001
  - mkl==2017.0.3=0
  - terminado==0.6=py36_0
  - libflac==1.3.1=hf484d3e_1002
  - python-levenshtein==0.12.2=pypi_0
  - werkzeug==0.14.1=pypi_0
  - pyspark==2.3.2=pypi_0
  - urllib3==1.26.9=pypi_0
  - bzip2==1.0.6=3
  - html5lib==0.9999999=py36_0
  - pywavelets==1.1.1=pypi_0
  - zeromq==4.1.5=0
  - pykerberos==1.2.1=pypi_0

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-aarch64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-aarch64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://conda.anaconda.org/conda-forge/linux-aarch64
  - https://conda.anaconda.org/conda-forge/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

May I know how can I install the packages successfully in the aarch64 architecture?

Last but not least, when I install package using pip install numpy, I got this error Illegal instruction (core dumped)

For this issue, may I know how can I solve this also in linux aarch64 architecture?

Answer 1

Very unlikely this will work for multiple reasons:

1. Package support off the major platforms (osx-64, linux-64, win-64) is sparse, especially further back in time. A concrete example is cudatoolkit, which only has linux-aarch64 builds starting with version 11.
2. Overgeneralized environment. The more packages included in an environment, the more difficult it becomes to solve it, and solving across platforms aggravates this problem. I would, for example, remove any Jupyter-related packages completely. In the future, try to plan ahead to have dedicated environments to specific projects, and only install the packages that are absolutely required.
3. Some packages are completely incompatible. For example mkl is architecture specific.

Nevertheless, if you want to attempt recreating an approximation of the environment, there are some options. First, one cannot achieve this with conda list --export - that simply does not handle environments that have packages from PyPI installed.

PyPI-centric Approach

Because so much of the environment is from PyPI, my first inclination is to recommend abandoning the Conda components and going a pip route. That is, use

pip list --format=freeze > requirements.txt

to capture the Python packages, then create a new environment with something like:

environment.yaml

name: foo
channels:
  - conda-forge
  - nodefaults
dependencies:
  - python=3.6
  - pip
  - pip:
    - -r requirements.txt

With both requirements.txt and environment.yaml in the same folder, the environment is created with

## "foo" is arbitrary - pick something descriptive
conda env create -n foo -f environment.yaml

Retaining some Conda packages

You could also try keeping some parts from Conda by mixing together a conda env export and the previous pip list. Specifically, export a minimal environment definition, with

conda env export --from-history > environment.yaml

Edit this file to include a specific version of Python, remove any packages that are not available for linux-aarch64 (like mkl), and add the pip: section, as above:

environment.yaml

#...
dependencies:
  - python=3.6
  # ...
  - pip
  - pip:
    - -r requirements.txt

This is then used with:

conda env create -n foo -f environment.yaml

Expect to iterate several times to discover what cannot be found for the platform. I would strongly recommend using mamba instead of Conda in order to minimize this solving time.