CODEWITHSUNDEEP
gcloudgcsfuse
logic1976
logic1976

Reputation: 581

Is writing files to a gcloud storage bucket supposed to be slow?

I'm using a gcloud storage bucket mounted to a VM instance with gcsfuse. I have no problems opening files and reading them when the files are stored on the storage bucket, but when I try to write files to the storage bucket it is enormously slow and when I say 'enormously' I mean at least 10 times slower if not 100 times. Is it supposed to be that way? If so, I guess I'm going to have to write files to a persistent disk, then upload the files to the storage bucket, then download the files to my personal computer from the storage bucket. Although the process will take the same amount of time, at least the psychological demoralization will not occur.

Upvotes: 0

Views: 2373

Answers (1)

sllopis
sllopis

Reputation: 2368

From Documentation:

Performance: Cloud Storage FUSE has much higher latency than a local file system. As such, throughput may be reduced when reading or writing one small file at a time. Using larger files and/or transferring multiple files at a time will help to increase throughput.

  • Individual I/O streams run approximately as fast as gsutil.
  • The gsutil rsync command can be particularly affected by latency because it reads and writes one file at a time. Using the top-level -m flag with the command is often faster.
  • Small random reads are slow due to latency to first byte (don't run a database over Cloud Storage FUSE!)
  • Random writes are done by reading in the whole blob, editing it locally, and writing the whole modified blob back to Cloud Storage. Small writes to large files work as expected, but are slow and expensive.

Optionally, please check out the gsutil tool or GCS Client Libraries, or even Storage Transfer Service since they may suit your needs better depending on your specific use case.

I hope this clarifies your concerns.

Upvotes: 1

Related Questions