CODEWITHSUNDEEP
python-3.xflaskgoogle-cloud-platformgoogle-cloud-run
Marcel
Marcel

Reputation: 319

Python are not able to find files on google cloud run in production

I pull data from an api and store and transform it locally. It worked local on my windows machine with docker but not in production on cloud run. I really dont know why this is happening.

app.py

...

try:
    os.mkdir(os.path.join(os.getcwd(), 'tmp'))
    os.mkdir(os.path.join(os.getcwd(), 'tmp', 'cpc'))
    os.mkdir(os.path.join(os.getcwd(), 'tmp', 'cpc', 'fin'))
    os.mkdir(os.path.join(os.getcwd(), 'tmp', 'cpc', 'raw'))
    os.mkdir(os.path.join(os.getcwd(), 'tmp', 'raw'))
    os.mkdir(os.path.join(os.getcwd(), 'tmp', 'transformed'))
except:
    print ("Creation of the directory failed")

...

loader.to_json(qcdp_final, os.path.join(os.getcwd(), 'tmp', 'raw', f'{last_week_monday}.json'))

...

load.py

...

def to_json(self, data, filename):
    self.data = data
    self.filename = filename
    with open(self.filename, 'w') as outfile:
        for row in self.data:
            json.dump(row, outfile)
            outfile.write('\n')
    outfile.close()

...

Stacktrace

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/raw/2020-09-07.json'
at to_json (load.py:44)
at download_data_from_api (extract.py:51)
at main (app.py:39)
at dispatch_request (/usr/local/lib/python3.8/site-packages/flask/app.py:1935)
at full_dispatch_request (/usr/local/lib/python3.8/site-packages/flask/app.py:1949)
at reraise (/usr/local/lib/python3.8/site-packages/flask/_compat.py:39)
at handle_user_exception (/usr/local/lib/python3.8/site-packages/flask/app.py:1820)
at full_dispatch_request (/usr/local/lib/python3.8/site-packages/flask/app.py:1951)
at wsgi_app (/usr/local/lib/python3.8/site-packages/flask/app.py:2446)

Upvotes: 2

Views: 828

Answers (2)

Dustin Ingram
Dustin Ingram

Reputation: 21580

The /tmp directory is not guaranteed to persist between container instances. If you want the files to be part of the docker image, you need to put them elsewhere.

Upvotes: 1

Marcel
Marcel

Reputation: 319

On Linux Systems it is forbidden to create folders like /tmp. This is reserved for the os. I specify a working directory in my Dockerfile like so WORKDIR /app and in this directory i create the tmp Folder. This do the trick.

Upvotes: 3

Related Questions