CODEWITHSUNDEEP
boto3python-poetrypython-s3fs
jtobelem
jtobelem

Reputation: 941

Resolving dependencies fails on boto3 and s3fs using poetry

I can install boto3, s3fs and pandas using :

pip install boto3 pandas s3fs

But it fails with poetry :

poetry add boto3 pandas s3fs

Here is the error :

Because no versions of s3fs match >2023.3.0,<2024.0.0
 and s3fs (2023.3.0) depends on aiobotocore (>=2.4.2,<2.5.0), s3fs (>=2023.3.0,<2024.0.0) requires aiobotocore (>=2.4.2,<2.5.0).
And because no versions of aiobotocore match >2.4.2,<2.5.0
 and aiobotocore (2.4.2) depends on botocore (>=1.27.59,<1.27.60), s3fs (>=2023.3.0,<2024.0.0) requires botocore (>=1.27.59,<1.27.60).
And because boto3 (1.26.91) depends on botocore (>=1.29.91,<1.30.0)
 and no versions of boto3 match >1.26.91,<2.0.0, s3fs (>=2023.3.0,<2024.0.0) is incompatible with boto3 (>=1.26.91,<2.0.0).
So, because engexploit-k8s-pod-operator-images depends on both boto3 (^1.26.91) and s3fs (^2023.3.0), version solving failed.

Upvotes: 12

Views: 7005

Answers (3)

BenPen
BenPen

Reputation: 392

After some more research, I think this is the bottom line: botocore and aiobotocore are very tightly linked. Because s3fs requires aiobotocore, there is exactly one version of botocore that matches a given s3fs version. So we are stuck with an older version of botocore if we need s3fs.

(Ref: https://github.com/fsspec/s3fs/issues/357#issuecomment-687376253)

SO, a good bet is to use venv to install a local copy of the modules in your project directory, use pip install -r requirements.txt to install the requested versions (module== or >= version number). (usage of venv is beyond the scope of this answer)

First create and install of the versions of modules except for s3fs, aiobotocore and botocore. Install these as above and resolve any version incompatibilities. Then install the desired s3fs version from the command line with pip install s3fs==version. It will report on the version mismatches between installed and desired, and hopefully they are only a point revision apart.

(This reminds me so much of dll incompatibilities way back in the pre .net ages.)

Upvotes: 0

Doug Hudgeon
Doug Hudgeon

Reputation: 1529

I don't know how much time I've spent over the years handling version conflicts between s3fs, aiobotocore and boto3... But it's alot! I've found adding these three dependencies to pyproject.toml in this order usually works for me:

s3fs = {extras = ["boto3"], version = ">=2023.12.0"}
boto3 = "*"
botocore = "*"

Note that I use whatever the latest version of s3fs is - which, at this time, is 2023.12.0.

Upvotes: 11

jtobelem
jtobelem

Reputation: 941

I was using s3fs through the pandas.read_csv method. I solved the question using only boto3, like in this answer.

Upvotes: 1

Related Questions