CODEWITHSUNDEEP
ruby-on-railsherokuupload
baash05
baash05

Reputation: 4516

Heroku... pushing without affecting long running methods

I have a long running import method in rails/heroku.
I'm importing a 10 meg csv and it takes a while..

While it's running a co-worker of mine makes a change to an unrelated view and does a git.push.

My import gets killed...

Right now; I can tell him not to do a git.push while I'm running the import. We have one client and all importing is done by the development team.

In the future we'll have dozens of clients all over the world. We can't control when they choose to do the import.

So my question is this... How do others protect against things like this? How do I make sure that jobs that are running don't get downed because I want to change something?

Upvotes: 0

Views: 94

Answers (1)

willglynn
willglynn

Reputation: 11520

The problem here is that your import method isn't robust.

Heroku kills your dyno when it receives a push. Heroku might also kill your dyno if it's using too much memory, or if it's been alive for a long time, or if they're rebalancing load, or if you scale the number of processes in that group. Moreover, that EC2 instance might just up and die for a variety of reasons, including power, network, storage, and human failures.

If it's important to you that imports always succeed, you need to build your application appropriately. Put the data you want to import in S3 temporarily, just in case the dyno that received the upload gets killed. Put a record in your database saying "this thing needs to be imported". When you're importing, import transactionally, so the import entirely succeeds or entirely fails.

Is that a lot of work? Maybe, or maybe not, depending on your choice of tools. (Sidekiq usually makes this easy for me.) However, this is your only real option.

The old school of thought was to make sure your infrastructure never fails, leading to SANs with multiply-redundant Fibre Channel uplinks going to multiply-redundant switches going to multiply-redundant storage arrays. This led to hot-swap CPUs, RAID for RAM, and other Enterprise Features™. But it turns out, even once you've done all that, your infrastructure still fails.

The new school of thought recognizes that failure is inevitable. Forget all that fancy stuff: buy commodity hardware in greater volumes, and build your applications to tolerate failures. That's the EC2 model, and that's the Heroku model. Design accordingly.

Upvotes: 3

Related Questions