CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-storagegoogle-cloud-rungcsfuse
Yousef Yassin
Yousef Yassin

Reputation: 1

Cloud Run Job GCSFuse Out of Memory

We're experiencing out of memory errors when running a cloud run job that only writes files to a mounted bucket. The job reads files from within the bucket, does some processing, and writes the output to the bucket.

I was under the impression that writing to the bucket would not consume all the job's allocated memory, but it doesn't seem to be the case, since I can see the memory consumption increasing throughout the job's lifetime (see image below). Here's the error we're getting:

time="01/08/2024 05:38:44.092331" severity=ERROR message="WriteFile: no space left on device, write /tmp/gcsfuse109880762: no space left on device"

I've run this job locally, on my machine, and the memory consumption is stable. Is the above a known issue? Does writing to a bucket consume the job's memory? If so, is there any cleanup I should be performing when writing to the mounted bucket from within a job?

I have also run with a GCSFuse mount locally, and could not reproduce the memory leak.

memory util

Upvotes: 0

Views: 351

Answers (2)

Ben K
Ben K

Reputation: 664

GCSFuse doesn't stream directly to GCS, it has to cache the file locally. That means you end up with a full copy of the file stored locally until you close the file, which since Cloud Run only has the in-memory file system means it consumes your instance's memory. On your local machine where it gets cached to your local storage, you probably have much more available space.

You can read more about GCSFuse's limitations here.

Upvotes: 1

Mr. Discuss
Mr. Discuss

Reputation: 435

The issue is not with the memory. As the error WriteFile: no space left on device suggests, you have run out of free space on your mounted disk to write the file.

Either you need to clean up the existing files you have, or the file you are trying to write is simply too large.

Upvotes: 0

Related Questions