CODEWITHSUNDEEP
pythongoogle-drive-api
Eric
Eric

Reputation: 244

Google Drive Python API: Uploading Large Files

I am writing a function to upload a file to Google Drive using the Python API client. It works for files up to 1 MB but does not work for a 10-MB file. When I try to upload a 10-MB file, I get an HTTP 400 error. Any help would be appreciated. Thanks.

Here is the output when I print the error:

An error occurred: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v3/files?alt=json&uploadType=resumable returned "Bad Request">

Here is the output when I print error.resp:

{'server': 'UploadServer', 'status': '400', 'x-guploader-uploadid': '...', 'content-type': 'application/json; charset=UTF-8', 'date': 'Mon, 26 Feb 2018 17:00:12 GMT', 'vary': 'Origin, X-Origin', 'alt-svc': 'hq=":443"; ma=2592000; quic=51303431; quic=51303339; quic=51303338; quic=51303337; quic=51303335,quic=":443"; ma=2592000; v="41,39,38,37,35"', 'content-length': '171'}

I'm unable to interpret this error. I have tried looking at the Google API Error Guide, but their explanation doesn't make sense to me, as all the parameters are the same as those in the requests with smaller files, which work.

Here is my code:

def insert_file_only(service, name, description, filename='', parent_id='root', mime_type=GoogleMimeTypes.PDF):
    """ Insert new file.

    Using documentation from Google Python API as a guide:
        https://developers.google.com/api-client-library/python/guide/media_upload

    Args:
        service: Drive API service instance.
        name: Name of the file to create, including the extension.
        description: Description of the file to insert.
        filename: Filename of the file to insert.
        parent_id: Parent folder's ID.
        mime_type: MIME type of the file to insert.
    Returns:
        Inserted file metadata if successful, None otherwise.
    """

    # Set the file meta data
    file_metadata = set_file_metadata(name, description, mime_type, parent_id)

    # Create media with correct chunk size
    if os.stat(filename).st_size <= 256*1024:
        media = MediaFileUpload(filename, mimetype=mime_type, resumable=True)
    else:
        media = MediaFileUpload(filename, mimetype=mime_type, chunksize=256*1024, resumable=True)

    file = None
    status = None
    start_from_beginning = True
    num_temp_errors = 0

    while file is None:
        try:
            if start_from_beginning:
                # Start from beginning
                logger.debug('Starting file upload')
                file = service.files().create(body=file_metadata, media_body=media).execute()
            else:
                # Upload next chunk
                logger.debug('Uploading next chunk')
                status, file = service.files().create(
                    body=file_metadata, media_body=media).next_chunk()
                if status:
                    logger.info('Uploaded {}%'.format(int(100*status.progress())))

        except errors.HttpError as error:
            logger.error('An error occurred: %s' % error)
            logger.error(error.resp)
            if error.resp.status in [404]:
                # Start the upload all over again
                start_from_beginning = True
            elif error.resp.status in [500, 502, 503, 504]:
                # Increment counter on number of temporary errors
                num_temp_errors += 1
                if num_temp_errors >= NUM_TEMP_ERROR_LIMIT:
                    return None
                # Call next chunk again
            else:
                return None

    permissions = assign_permissions(file, service)
    return file

UPDATE
I tried using a simpler pattern, taking the advice from @StefanE. However, I still get an HTML 400 error for files over 1 MB. New code looks like this:

request = service.files().create(body=file_metadata, media_body=media)
response = None
while response is None:
    status, response = request.next_chunk()
    if status:
        logger.info('Uploaded {}%'.format(int(100*status.progress()))

UPDATE 2
I found that the issue is conversion of the file into a Google Document, not uploading it. I'm trying to upload an HTML file and convert it into a Google Doc. This works for files less than ~2 MB. When I only upload the HTML file but not try to convert it, I don't get the abovementioned error. Looks like this corresponds with the limit on this page. I don't know if this limit can be increased.

Upvotes: 3

Views: 2933

Answers (1)

StefanE
StefanE

Reputation: 7630

I see some issues with your code.

First you have a while loop to continue as long file is None and the first thing you do is to set the value of file. i.e it will only loop once.

Secondly you got variable start_from_beginning but that is never set to False anywhere in the code, the else part of the statement will never be executed.

Looking at the Googles documentation their sample code looks a lot more straight forward:

media = MediaFileUpload('pig.png', mimetype='image/png', resumable=True)
request = farm.animals().insert(media_body=media, body={'name': 'Pig'})
response = None
while response is None:
  status, response = request.next_chunk()
  if status:
    print "Uploaded %d%%." % int(status.progress() * 100)
print "Upload Complete!"

Here you loop on while response is None which will be None until finished with the upload.

Upvotes: 1

Related Questions