CODEWITHSUNDEEP
pythonamazon-web-servicesaws-glue
Vzzarr
Vzzarr

Reputation: 5700

AWS Glue Python Shell download_and_install(args.extra_py_files) - download_from_s3(s3_file_path, local_file_path)

I am using AWS Glue Python Shell type. The Job fails with the error Command failed with exit code 1. If I try to see Logs are not available for the specific run. The only message I can get from Error logs is this error message:

Traceback (most recent call last):
  File "/tmp/runscript.py", line 112, in <module>
    download_and_install(args.extra_py_files)
  File "/tmp/runscript.py", line 55, in download_and_install
    download_from_s3(s3_file_path, local_file_path)
  File "/tmp/runscript.py", line 80, in download_from_s3
    s3.download_file(bucket_name, s3_key, new_file_path)
  File "/usr/local/lib/python3.6/site-packages/boto3/s3/inject.py", line 172, in download_file
    extra_args=ExtraArgs, callback=Callback)
  File "/usr/local/lib/python3.6/site-packages/boto3/s3/transfer.py", line 307, in download_file
    future.result()
  File "/usr/local/lib/python3.6/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.6/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/usr/local/lib/python3.6/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/s3transfer/download.py", line 345, in _submit
    **transfer_future.meta.call_args.extra_args
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

which does not give me much information on what is wrong (all functions / python files reported in the error message are totally unrelated to the script I submitted). Any help?

Upvotes: 1

Views: 1206

Answers (1)

Vzzarr
Vzzarr

Reputation: 5700

It turns out I was importing incorrectly libraries in the field Python lib path, referencing a file that does not exist in S3.

The error message was not very helpful... I hope my experience can help other users

Upvotes: 3

Related Questions