CODEWITHSUNDEEP
ruby-on-railsrubyamazon-web-servicesherokuamazon-s3
jdesilvio
jdesilvio

Reputation: 1854

The Mystical Ephemeral File System of Heroku is Not Letting Me Get Files from S3

So, I've been wrestling with this issue for days...I need to get a file from S3 and write it to a directory in my Rails app on Heroku. I must have a misunderstanding of the ephemeral file system on Heroku because I can't figure out why it's not working.

I am running s3.bucket('bucket').object('file.csv').get(response_target: 'file.csv') to get a file from S3 and write it to my app. Initially I just wrote a .rb to do this and ran it using the Heroku Scheduler, but to no avail. I then turned the script into a rake task and ran that on the scheduler, again, to no avail. I am able to run both the .rb script and the rake task flawlessly in my dev environment.

After reading this and this on how the ephemeral file system works, I am thinking that the task actually is working, but the file gets destroyed (or is actually there but I can't see it?) when I use ls in heroku run bash.

Can someone please explain what is going on to me? If my efforts to get a file from S3 written to my app on Heroku are futile? And if there are any alternatives?

If I can't figure it out after this then I am going to set up my own env in EC2.

Upvotes: 1

Views: 764

Answers (1)

matt
matt

Reputation: 79803

With Heroku you don’t have a single app running, rather you have several dynos each with a copy of your code and running some aspect of your app, and each independent from the others. In particular each dyno’s file system is separate from the others.

In your case you push your app and this creates one (or maybe more) web dynos, which runs your Rails app – handling web requests.

You also have a scheduled task using the Heroku Scheduler that downloads the file. When this runs a new one-off dyno is created and the file is downloaded into that dyno’s file system. When the task is completed that dyno, along with the downloaded file, is discarded.

When you run heroku run bash you create yet another one-off dyno, and obviously the file isn’t in that dyno’s filesystem.

The solution will depend on exactly what you are trying to do, but one suggestion would be to put the data from the file into your database where the other dynos can access it easily.

Upvotes: 3

Related Questions