CODEWITHSUNDEEP
rubymultithreadingparallel-processingrake
Jack
Jack

Reputation: 497

Ruby rake tasks thread optimization

I have some rake tasks that need to run every hour approximately, each one of them has to contact a web site, download a file and import the file into my db.

I was wondering what the best approach would be in terms of efficiency and speed of execution. Am I correct if I think that making a thread for each task will save me some time? By doing so I should be able to minimize the time spent waiting for the server response to that of the slowest one among the threads.

Once downloaded the files, I was thinking of using gem "parallel" for importing data into the db from large csv and xml files.

Please let me know should you have any suggestions.

Upvotes: 3

Views: 533

Answers (1)

Vasfed
Vasfed

Reputation: 18454

When number of such files is low, you do not care for order of execution and can afford some extra memory - simpliest solution is just to run them in different processes by cron (for example - gem 'whenever').

If there're more - use some http gems for parallel downloading - typhoeus, curb, em-http-request etc

Upvotes: 3

Related Questions