CODEWITHSUNDEEP
pythonamazon-s3aws-glue
j raj
j raj

Reputation: 169

Import failure of s3fs library in AWS Glue

AWS glue is not importing s3fs module

import s3fs

I expect the library to be imported but AWS glue says

ImportError : No module named s3fs

Upvotes: 6

Views: 12344

Answers (2)

Zcauchon
Zcauchon

Reputation: 68

s3fs is only included in Glue 2.0 and up. If you are trying to use this in a Python shell Glue job which uses Glue 1.0, you'll have to provide the whl file for s3fs as Sherlock mentioned above.

Here is a list of the default packages for Python Shell jobs https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#python-shell-supported-library

Upvotes: 1

Sherlock
Sherlock

Reputation: 5627

AWS Glue jobs come with some common libraries pre installed but for anything more than that you need to download the .whl for the library from pypi, which in the case of s3fs can be found here.

Once you have that, upload it to an s3 bucket, eg. s3://my-libraries/ and reference it in the Python library path field in the console.

enter image description here

This will prompt Glue to install the libraries within this bucket prior to running the script. Note that only pure python libraries are support currently.

Upvotes: 6

Related Questions