CODEWITHSUNDEEP
shelldebiangoogle-cloud-storagegoogle-compute-engine
Fabio Benedetti
Fabio Benedetti

Reputation: 317

Instance of Google Compute Engine freezes trying to upload files on Google Cloud Storage

I have wrote this shell script that download archives from a url list, decompresses them and finally moves them in a Cloud Storage bucket.

#!/bin/bash
# declare STRING variable

for iurl in $(cat ./html-rdfa.list); do
    filename=$(basename "$iurl")
    file="${filename%.*}"
    if gsutil ls gs://rdfa/$file; then
        echo "yes"
    else
        wget $iurl
        gunzip $filename
        gsutil cp -n $file gs://rdfa
        rm $file
        sleep 2
    fi
done

html-rdfa.list contains the url list. The instance is created using the debian 7 image provided by gooogle.

The script run correctly for the first 5 or 6 files, but then the instance freezes and i have to delete the instance. The ram or the disk of the instance are not full when it freezes.

I think the problem is caused by the command gsutil cp, but it is strange that CPU load is practically 0 and also the RAM is free but it is impossibilo to use the instance without restarting them.

Upvotes: 0

Views: 1492

Answers (1)

Brian Dorsey
Brian Dorsey

Reputation: 4688

Are you writing the temporary files to the default 10GB root disk? If so, you may be running into the Persistent Disk throughput caps. To see if this is the case, create a new Persistent Disk, then mount it as a data disk and use that disk for the temporary files. Consider starting with ~200GB disk and see if that is enough throughput for your workload. Also, see the docs on Persistent Disk performance.

Upvotes: 2

Related Questions