CODEWITHSUNDEEP
python-3.xboto3
Agost Biro
Agost Biro

Reputation: 2849

How to check if boto3 S3.Client.upload_fileobj succeeded?

I want to save the result of a long running job on S3. The job is implemented in Python, so I'm using boto3. The user guide says to use S3.Client.upload_fileobj for this purpose which works fine, except I can't figure out how to check if the upload has succeeded. According to the documentation, the method doesn't return anything and doesn't raise an error. The Callback param seems to be intended for progress tracking instead of error checking. It is also unclear if the method call is synchronous or asynchronous.

If the upload failed for any reason, I would like to save the contents to the disk and log an error. So my question is: How can I check if a boto3 S3.Client.upload_fileobj call succeeded and do some error handling if it failed?

Upvotes: 21

Views: 14959

Answers (3)

Mark
Mark

Reputation: 3557

There is a wait_until_exists() helper function that seems to be for this purpose in the boto3.resource object.

This is how we are using it:

s3_client.upload_fileobj(file, BUCKET_NAME, file_path)
s3_resource.Object(BUCKET_NAME, file_path).wait_until_exists()

Upvotes: 4

reubano
reubano

Reputation: 5373

I use a combination of head_object and wait_until_exists.

import boto3

from botocore.exceptions import ClientError, WaiterError

session = boto3.Session()
s3_client = session.client('s3')
s3_resource = session.resource('s3')


def upload_src(src, filename, bucketName):
    success = False

    try:
        bucket = s3_resource.Bucket(bucketName)
    except ClientError as e:
        bucket = None

    try:
        # In case filename already exists, get current etag to check if the 
        # contents change after upload
        head = s3_client.head_object(Bucket=bucketName, Key=filename)
    except ClientError:
        etag = ''
    else:
        etag = head['ETag'].strip('"')

    try:
        s3_obj = bucket.Object(filename)
    except ClientError, AttributeError:
        s3_obj = None

    try:
        s3_obj.upload_fileobj(src)
    except ClientError, AttributeError:
        pass
    else:
        try:
            s3_obj.wait_until_exists(IfNoneMatch=etag)
        except WaiterError as e:
            pass
        else:
            head = s3_client.head_object(Bucket=bucketName, Key=filename)
            success = head['ContentLength']

    return success

Upvotes: 15

anoop-khandelwal
anoop-khandelwal

Reputation: 3860

I would recommend you to perform the following operations-

  try:
     response = upload_fileobj()
  except Exception as e:
     save the contents to the disk and log an error.
  if response is None:  

     polling after every 10s to check if the file uploaded successfully or not using **head_object()** function..
     If you got the success response from head_object :
          break
     If you got error in accessing the object:
        save the contents to the disk and log an error.

So , basically do poll using head_object()

Upvotes: 3

Related Questions