CODEWITHSUNDEEP
pythonamazon-web-servicesaws-glue
sam
sam

Reputation: 1896

Python AWS Glue log says "Considering file without prefix as a python extra file" for uploaded python zip packages

In AWS Glue, for a simple pandas job of reading data in XLSX and writing to CSV. I have a small code. As per the Python Glue instructions, I have zipped the required libraries and provided the as packages to Glue Job while execution.

Question: What do the following logs convey?

Considering file without prefix as a python extra file s3://raw-data/sampath/scripts/s3fs/fsspec.zip
Considering file without prefix as a python extra file s3://raw-data/sampath/scripts/s3fs/jmespath.zip
Considering file without prefix as a python extra file s3://raw-data/sampath/scripts/s3fs/s3fs.zip
....

please elaborate with an example?

Upvotes: 1

Views: 3831

Answers (1)

Sandeep Fatangare
Sandeep Fatangare

Reputation: 2144

In python shell jobs, you should add external libraries in egg file and not zip file. Zip file is for Spark job.

I also wrote small shell script to deploy python shell job without manual steps to create egg file and upload to s3 and deploy via cloudformation. Script does all automatically. You may find code at https://github.com/fatangare/aws-python-shell-deploy. Script will take csv file and convert it into excel file using pandas and xlswriter libraries.

Upvotes: 5

Related Questions