CODEWITHSUNDEEP
pythonconda
jamiet
jamiet

Reputation: 12304

How do I interpret this python dependency tree?

We are using conda to maintain a python environment and I'd like to understand why google-cloud-bigquery==1.22.0 is being installed when the latest available version is https://pypi.org/project/google-cloud-bigquery/2.16.1/ and the latest vaailable version on conda-forge (https://anaconda.org/conda-forge/google-cloud-bigquery) is 2.15.0

Here's a Dockerfile that builds our conda environment:

FROM debian:stretch-slim

RUN apt-get update && apt-get install curl gnupg -y && rm -rf /var/lib/apt/lists/*

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
        install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
        gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring \
        --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
        echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
        > /etc/apt/sources.list.d/conda.list

WORKDIR /tmp
RUN MINICONDA_VERSION=4.9.2 && \
        CONDA_VERSION='4.9.*' && \
        CONDA_DIR=/opt/conda && \
        curl -O https://repo.anaconda.com/miniconda/Miniconda3-py37_${MINICONDA_VERSION}-Linux-x86_64.sh && \
        /bin/bash Miniconda3-py37_${MINICONDA_VERSION}-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
        rm Miniconda3-py37_${MINICONDA_VERSION}-Linux-x86_64.sh && \
        $CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
        $CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
        $CONDA_DIR/bin/conda config --system --remove channels defaults && \
        $CONDA_DIR/bin/conda config --system --add channels main && \
        $CONDA_DIR/bin/conda config --system --set env_prompt '({name}) ' && \
        $CONDA_DIR/bin/conda config --system --append envs_dirs /opt/conda/envs/ && \
        $CONDA_DIR/bin/conda config --system --append pkgs_dirs /opt/conda/pkgs/ && \
        $CONDA_DIR/bin/conda update --quiet --yes --all conda="${CONDA_VERSION}" && \
        $CONDA_DIR/bin/conda config --system --append channels conda-forge  && \
        $CONDA_DIR/bin/conda create -n py3 python=3.8

RUN bash -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && conda install \
                invoke \
                apache-beam \
                sh \
                pytest \
                pytest-xdist \
                ipython \
                behave \
                black \
                pylint \
                flake8 \
                jinja2 \
                tenacity \
                responses \
                tqdm \
                google-api-python-client \
                google-auth-oauthlib \
                google-cloud-monitoring \
                google-cloud-bigquery \
                google-cloud-storage \
                google-cloud-pubsub \
                google-cloud-secret-manager \
                ipdb \
                rope \
                pipdeptree"

I build it using docker build . -t conda-env and then use pipdeptree inside the container to give me my dependency tree for google-cloud-bigquery:

docker run \
  --rm \
  --entrypoint bash \
  conda-env \
  -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && pipdeptree --packages google-cloud-bigquery"

which gives me this:

google-cloud-bigquery==1.22.0
  - google-cloud-core [required: >=1.0.3,<2.0dev, installed: 1.6.0]
    - google-api-core [required: >=1.21.0,<2.0.0dev, installed: 1.25.1]
      - google-auth [required: >=1.21.1,<2.0dev, installed: 1.28.1]
        - cachetools [required: >=2.0.0,<5.0, installed: 4.2.1]
        - pyasn1-modules [required: >=0.2.1, installed: 0.2.8]
          - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
        - rsa [required: >=3.1.4,<5, installed: 4.7.2]
          - pyasn1 [required: >=0.1.3, installed: 0.4.8]
        - setuptools [required: >=40.3.0, installed: 52.0.0.post20210125]
        - six [required: >=1.9.0, installed: 1.15.0]
      - googleapis-common-protos [required: >=1.6.0,<2.0dev, installed: 1.53.0]
        - protobuf [required: >=3.12.0, installed: 3.14.0]
          - six [required: >=1.9, installed: 1.15.0]
      - protobuf [required: >=3.12.0, installed: 3.14.0]
        - six [required: >=1.9, installed: 1.15.0]
      - pytz [required: Any, installed: 2021.1]
      - requests [required: >=2.18.0,<3.0.0dev, installed: 2.25.1]
        - certifi [required: >=2017.4.17, installed: 2020.12.5]
        - chardet [required: >=3.0.2,<5, installed: 3.0.4]
        - idna [required: >=2.5,<3, installed: 2.10]
        - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.4]
      - setuptools [required: >=40.3.0, installed: 52.0.0.post20210125]
      - six [required: >=1.13.0, installed: 1.15.0]
    - google-auth [required: >=1.24.0,<2.0dev, installed: 1.28.1]
      - cachetools [required: >=2.0.0,<5.0, installed: 4.2.1]
      - pyasn1-modules [required: >=0.2.1, installed: 0.2.8]
        - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
      - rsa [required: >=3.1.4,<5, installed: 4.7.2]
        - pyasn1 [required: >=0.1.3, installed: 0.4.8]
      - setuptools [required: >=40.3.0, installed: 52.0.0.post20210125]
      - six [required: >=1.9.0, installed: 1.15.0]
    - six [required: >=1.12.0, installed: 1.15.0]
  - google-resumable-media [required: >=0.3.1,<0.6.0dev,!=0.4.0, installed: 0.5.1]
    - six [required: Any, installed: 1.15.0]
  - protobuf [required: >=3.6.0, installed: 3.14.0]
    - six [required: >=1.9, installed: 1.15.0]

I have to hold my hand up and admit I simply don't know how to interpret that. For example, does this:

google-cloud-bigquery==1.22.0
  - google-cloud-core [required: >=1.0.3,<2.0dev, installed: 1.6.0]

mean that google-cloud-core is forcing google-cloud-bigquery to be ">=1.0.3,<2.0dev" or does it mean google-cloud-core is being forced by something else to be ">=1.0.3,<2.0dev". I basically don't understand that information being presented to me so I'm hoping someone can enlighten me. better still, if someone can tell me what I can do to get a later version of google-cloud-bigquery installed I'd be very grateful, because there are known bugs in 1.22.0.

Upvotes: 1

Views: 267

Answers (1)

thetillhoff
thetillhoff

Reputation: 554

To answer your last question first:

google-cloud-bigquery==1.22.0
  - google-cloud-core [required: >=1.0.3,<2.0dev, installed: 1.6.0]

means, that google-cloud-bigquery is installed with version 1.22.0, which requires google-cloud-core to be installed with a version between 1.0.3 and 2.0 and you have version 1.6.0 of it installed.

To check the constraints of google-cloud-bigquery - which you are probably trying to do, add the --reverse flag like that: pipdeptree --reverse --packages google-cloud-bigquery. The output isn't useful though, because "you're looking at the wrong side of the tree":

Warning!!! Possibly conflicting dependencies found:
* pylint==2.7.4
 - astroid [required: >=2.5.2,<2.7, installed: 2.5]
* flake8==3.9.0
 - pycodestyle [required: >=2.7.0,<2.8.0, installed: 2.6.0]
 - pyflakes [required: >=2.3.0,<2.4.0, installed: 2.2.0]
------------------------------------------------------------------------
google-cloud-bigquery==1.22.0

So to see the actual restraint, run pipdeptree --reverse and look for google-cloud-bigquery. Then you'll find, that urllib3 in version 1.26.4 constraints requests to version 2.25.1, which constraints google-api-core to 1.25.1, which constraints google-cloud-core to 1.6.0, which constraints google-cloud-bigquery to 1.22.0.

If I had to guess, I'd say that one of those mentioned packages is already installed (EDIT: even before you're installing your specific packages) in their respective version, which leads to what you're seeing.

I ran pip install google-cloud-bigquery --upgrade on top of your build, which worked perfectly fine, so you could either run that at the end or just upgrade all packages before installing your specific stuff (which I'd personally recommend, updating your base is always a good idea. That being said, if your own constraints are "too old", that wouldn't work and you should fall back to upgrading after you've installed your specific stuff).

Upvotes: 3

Related Questions