Google Cloud Storage: Setting incorrect MIME-type

Question

I have a Node.js server running on a Google Compute Engine virtual instance. The server streams incoming files to Google Cloud Storage GCS. My code is here: Node.js stream upload directly to Google Cloud Storage

I'm passing Content-Type in the XML headers and it's working just fine for image/jpeg MIME-types, but for video/mp4 GCS is writing files as application/octet-stream.

There's not much to this, so I'm totally at a loss for what could be wrong ... any ideas are welcome!

Update/Solution

The problem was due to the fact that the multiparty module was creating content-type: octet-stream headers on the 'part' object that I was passing into the pipe to GCS. This caused GCS to receive two content-types, of which the octet part was last. As a result, GCS was using this for the inbound file.

Answer 1

When migrating from Amazon S3 to Google Cloud Platform (GCP), I encountered similar challenges. To address these, I implemented Resumable Upload for uploading directory contents to GCP. Here’s the approach:

Dependencies and Setup:

Annotations:

@Override: Indicates that the method overrides a parent method.

@SneakyThrows: A Lombok annotation that eliminates the need to explicitly declare checked exceptions.

Method Parameters:

File digitalContentDirectory: The directory containing files to be uploaded.

boolean includeSubdirectories: A flag to determine whether subdirectories should be included.

String key: The base path (prefix) for uploaded objects in the GCS bucket.

Google Cloud Storage Client Initialization

Storage storage = StorageOptions.getDefaultInstance().getService();

This initializes the GCS client to interact with your bucket.

Directory Traversal

• The Files.walk method is used to traverse the directory:
  • If includeSubdirectories is true, it recursively includes all files and subdirectories.
  • Otherwise, it processes only the root directory.
• Filters out non-regular files (e.g., directories, symbolic links).

Processing and Uploading Files

1. MIME Type Detection

• Uses Files.probeContentType(filePath) to determine the file's MIME type.
• Defaults to application/octet-stream if the MIME type cannot be detected.

2. Relative Path Calculation

• Calculates the file’s relative path based on the root directory using:

  Path relativePath = rootPath.relativize(filePath);

• Ensures that uploaded objects retain the original directory structure.

3. Object Name Construction

• Constructs the object name in GCS by combining the key with the relative path.

• Replaces backslashes () with forward slashes (/) to comply with GCS naming conventions.

Resumable Upload to GCS

1. BlobInfo Creation

• Creates metadata for each file (BlobInfo), including:
• The destination path in the bucket.
• The file’s MIME type.

2. File Upload

• Opens a FileChannel to read the file in chunks (256 KB buffer).
• Writes to GCS using a WriteChannel provided by the GCS SDK.
• Logs successful uploads.

3. Error Handling

• Wraps file upload logic in a try-catch block:
• Logs any exceptions encountered.
• Rethrows exceptions to ensure visibility in failure scenarios.

Return URL

Returns a public URL for the uploaded files, constructed as:

https://storage.googleapis.com/<bucketName>/<key>

This approach worked seamlessly for my migration project, enabling efficient uploads with minimal disruptions.

Answer 2

this worked for me:

  const metadata = {
  contentType: "video/mp4",
  metadata: {
    custom: "abc"
  }
};
await bucket.file(filePath).setMetadata(metadata);

Answer 3

A solution that worked for me in a similar situation is below. TLDR: Save video from web app to GCS with content type video/mp4 instead of application/stream.

Here is the situation. You want to record video in the browser and save it to Google Cloud Storage with a content type set to video/mp4 instead of application/octet-stream. User records video and clicks button to send video file to your server for saving. After sending the video file from the client to your server, the server sends the video file to Google Cloud Storage for saving.

You successfully save the video to Google Cloud Storage and by default GCS assigns a content type of application/octet-stream to the video.

To assign a content type video/mp4 instead of application/octet-stream, here is some server-side Python code that works.

storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
blob.upload_from_file(file_obj, rewind=True)
blob.content_type = 'video/mp4'
blob.patch()

Here are some links that might help.

https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-python
https://stackoverflow.com/a/33320634/19829260
https://stackoverflow.com/a/64274097/19829260

NOTE: at the time of this writing, the Google Docs about editing metadata don't work for me because they say to set metadata but metadata seems to be read-only (see SO post https://stackoverflow.com/a/33320634/19829260)

https://cloud.google.com/storage/docs/viewing-editing-metadata#edit

Answer 4

Ok, looking at your HTTP request and response it seems like content-type is specified in the URL returned as part of the initial HTTP request. The initial HTTP request should return the endpoint which can be used to upload the file. I'm not sure why that is specified there but looking at the documentation (https://developers.google.com/storage/docs/json_api/v1/how-tos/upload - start a resumable session) it says that X-Upload-Content-Type needs to be specified, along some other headers. This doesn't seem to be specified in HTTP requests that were mentioned above. There might be an issue with the library used but the returned endpoint does not look as what is specified in the documentation.

Have a look at https://developers.google.com/storage/docs/json_api/v1/how-tos/upload, "Example: Resumable session initiation request" and see if you still have the same issue if you specify the same headers as suggested there.

Answer 5

Google Cloud Storage is content-type agnostic, i.e., it treats any kind of content in the same way (videos, music, zip files, documents, you name it).

---

But just to give some idea,

First I believe that the video () you are uploading is more or less size after it being uploded. so , it falls in application/<sub type>. (similar to section 3.3 of RFC 4337)

To make this correct, I believe you need to fight with storing mp4 metadata before and after the file being uploaded.

please let us know of your solution.