CODEWITHSUNDEEP
pythongoogle-cloud-storagepre-signed-url
Luv33preet
Luv33preet

Reputation: 1867

Uploading File to GCS - Metadata part is too large

We are trying to upload files to GCS using signed urls. The client asks for a signed url to upload a file and our server sends a signed url to the client. The client then uploads the file using that signed url. This is working fine for cases where file size < 2MB. But the limit for the object size in GCS is 5TB.

Below is the code used to generate the signed url


blob = Blob.from_string(filepath, client=gcp_storage_client)
blob.generate_signed_url(version="v4", expiration=datetime.timedelta(minutes=5), method="POST")

We are getting the response "Metadata part is too large". But we are not adding any metadata to the file. Please check the screenshot from Postman below.

What am I doing wrong here? and how can I fix it?

enter image description here

Upvotes: 1

Views: 1215

Answers (1)

Brandon Yarbrough
Brandon Yarbrough

Reputation: 38389

GCS has several mechanisms to upload objects. The one you're using is the HTML form "POST object" mechanism. POST uploads do not support signed URLs. They use signed policy documents instead, which are a bit more complicated but have some extra capabilities.

If your client is using not a human being using an HTML form, I suggest using PUT object for uploads. It's simpler, and it supports signed URLs. It would look like this:

PUT /OBJECT_NAME??X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=whatnot HTTP/1.1
Host: BUCKET_NAME.storage.googleapis.com
Date: DATE
Content-Length: FILE_LENGTH
Content-Type: FILE_MIME_TYPE
Content-MD5: MD5_DIGEST
x-goog-storage-class: DESIRED_STORAGE_CLASS

<file goes here>

Upvotes: 2

Related Questions