CODEWITHSUNDEEP
ruby-on-railsjsonmongodbmongoid
MervS
MervS

Reputation: 5902

Rendering large database record set as JSON in rails

Is there a way to present a large record set in JSON format without consuming large amount of memory? For example I have the following query:

...
records = Records.where(query)
respond_to do |format|
  format.html
  format.json  { render :json => records.to_json }
end

There are times that records will contain thousands of entries and JSON is strictly used for getting the data without using pagination and such data must fit inside the memory for it to be returned. A Record entry will also contain a lot of fields (I am using MongoDB/Mongoid) and it is necessary to include such fields.

Upvotes: 1

Views: 1222

Answers (1)

ck3g
ck3g

Reputation: 5929

It's almost always a bad idea to return every resource in a database.

You can respond with limited set of results and provide total number of records.

For example:

{
  total: 503
  records: [
    { id: 1 },
    { id: 2 }
  ]
}

And add possibility to use limit and offset parameters to iterate through all pages.

There is chapter named Pagination and partial response in free e-book Web API Design describes it.

I would recommend you to read that book. It contains only 30 pages.

upd: In your case you are able to paginate results using limit and skip mongoid's methods.

records = Records.where(query).limit(params[:limit]).skip(params[:offset])
respond_to do |format|
  format.html
  format.json  { render :json => { total: records.count, records: records }.to_json }
end

Upvotes: 1

Related Questions