CODEWITHSUNDEEP
pythontensorflowcondabrotli
Gameriker
Gameriker

Reputation: 55

Conda broken dependencies

I'm installing dependencies from three files with script:

#!/bin/bash
source /miniconda/bin/activate base # does not create .bashrc entry
conda create -p $APP_VENV -y
mamba env update -p $APP_VENV --file $APP_PATH/requirements.yaml
mamba env update -p $APP_VENV --file $WORK_PATH/external/spacetimeformer/requirements.yaml
mamba env update -p $APP_VENV --file $COMMON_PATH/requirements.yaml

and after installation of app_requirements there are following problems

1.Brotli lib Undefined Symbol:

ImportError: /workspace/.venv/lib/python3.10/site-packages/pyarrow/../../.././libbrotlidec.so.1: undefined symbol: BrotliSharedDictionaryDestroyInstance

2. Tensorflow cannot access cuda device, however pytorch has the access.

3. Cannot import keras from tensorflow despite that it is 2.16.1

Here are the aforementioned files with requirements

# app_requirements.yaml
channels:
  - defaults
  - conda-forge    
dependencies:
  - python=3.10.14
  - conda-forge::overrides
  - conda-forge::statsmodels
  - conda-forge::tensorflow=2.16.1 # somehow not always includes keras
  - conda-forge::keras
  - conda-forge::prettytable
  - conda-forge::pmdarima 
  - conda-forge::xgboost

# common_requirements.yaml
channels:
  - defaults
  - conda-forge 
dependencies:
  - python=3.10.14
  - numpy
  - pandas
  - certifi
  - requests
  - six
  - toml
  - urllib3
  - python-dateutil
  - pytz
  - conda-forge::pandas
  - conda-forge::toolz
  - conda-forge::pandas_market_calendars
  - conda-forge::korean_lunar_calendar
  - conda-forge::charset-normalizer
  - conda-forge::idna
  - conda-forge::setuptools
  - conda-forge::tzdata
  - conda-forge::wheel
  - conda-forge::attrs
  - conda-forge::referencing
  - conda-forge::jsonschema-specifications
  - conda-forge::rpds-py
  - conda-forge::overrides
  - conda-forge::jsonref
  - conda-forge::jsonschema
  - conda-forge::pymongo
  - conda-forge::pymongoarrow
  - conda-forge::ipykernel
  - conda-forge::inflection

  - pip
  - pip:
      - confluent-kafka==2.3.0

# external_reuqirements.yaml
channels:
  - defaults
  - conda-forge    
dependencies:
  - python=3.10.14
  - cython>=0.22
  - numpy>=1.15.4
  - pandas>=1.0.4
  - matplotlib>=2.0.0
  - convertdate>=2.1.2
  - python-dateutil>=2.8.0
  - tqdm>=4.36.1
  - seaborn
  - chardet
  - opt_einsum
  - conda-forge::omegaconf
  - conda-forge::einops
  - conda-forge::opencv
  - conda-forge::scikit-learn
  - conda-forge::netcdf4
  - conda-forge::pystan   
  # - conda-forge::wandb   
  # - pystan=2.19.1.1
  - conda-forge::torchvision
  - conda-forge::torchmetrics=0.5.1
  - conda-forge::pytorch-lightning=1.6
  - conda-forge::performer-pytorch
  - conda-forge::nystrom-attention
  - conda-forge::cmdstanpy=0.9.68

Do you have any idea what might be breaking this conda env? Thank you in advance

Upvotes: 0

Views: 81

Answers (1)

James
James

Reputation: 36746

For 1: this could be an issue with the version of snappy that pyarrow is using (snappy uses brotli). You can try pinning the version to pre 1.2.:

  - snappy=1.1.10

For 2: you likely have installed the CPU build of tensorflow from conda-forge. You can pin to the CUDA builds using:

  - conda-forge::tensorflow=2.16.1=cuda*

For 3: not sure how to help if you don't give the code you ran and the traceback.

Upvotes: 0

Related Questions