CODEWITHSUNDEEP
ruby-on-railsruby
avellable
avellable

Reputation: 1395

What is the most efficient way to store temp data for processing?

I am writing an application in Ruby, which collect huge amount of data from API calls and stores it in a file. After that it processes it one by one. I was wondering if there is a way better than this to achieve the same?

Note: I want to process the records one by one by storing all of them locally because they may change during API calls.

Upvotes: 2

Views: 675

Answers (1)

CDub
CDub

Reputation: 13354

I would look at storing the information in an in-memory key/value store (such as memcached or redis). If you use an in-memory key/value store, you can update information based on subsequent API calls rather than having multiple records in a file which represent the same data, just with different values.

Keep in mind, however, if your data is significantly large, you may run out of memory. That said, if you are into the gigabytes of data, the way you have implemented your solution may be the best route to take.

Upvotes: 1

Related Questions