CODEWITHSUNDEEP
jsonruby-on-rails-3
donald
donald

Reputation: 23747

Rails 3: How to return a big JSON document

I want to return about 90k items in a JSON document but I'm getting this error when I make the call:

Timeout::Error in ApisController#api_b

time's up!
Rails.root: /root/api_b

I am simply running "rails s" with the default rails server.

What's the way to make this work and return the document?

Thanks

  @bs.each do |a|
          puts "dentro do bs.each"
          @final << {  :Email => a['headers']['to'], :At => a['date'], :subject => a['headers']['subject'], :Type => a['headers']['status'], :Message_id => a['headers']['message_id'] }
        end

Being @bs the BSON object from MongoDB. The timeout is in "@final << ..."

Upvotes: 1

Views: 1425

Answers (1)

christiangeek
christiangeek

Reputation: 619

If you are experiencing timeouts from rails and it is possible to cache the data (e.g. the data changes infrequently), I would generate the response in the background using resque or delayed_job and than have Rails dump that to the client. Or if the data cannot be cached, use a lightweight Rack handler like Sinatra and Metal to generate the responses.

Edited to reflect sample data

I was able to run the following code in a Rails 3.0.9 instance against a high performance Mongo 1.8.4 instance. I was using Mongo 1.3.1, bson_ext 1.3.1, webrick 1.3.1 and Ruby 1.9.2p180 x64. It did not time out but it took some time to load. My sample Mongo DB has 100k records and contains no indexes. 

before_filter :profile_start
after_filter :profile_end
  def index
    db   = @conn['sample-dbs']
    collection = db['email-test']

    @final = []
    @bs = collection.find({})
    @bs.each do |a|
      puts "dentro do bs.each"
      @final << {  :Email => a['headers']['to'], :At => a['date'], :subject => a['headers']['subject'], :Type => a['headers']['status'], :Message_id => a['headers']['message_id'] }
    end
    render :json => @final
  end

  private
    def profile_start
      RubyProf.start
    end

    def profile_end
      RubyProf::FlatPrinter.new(RubyProf.stop).print
    end

A more efficient way to dump out the records would be

@bs = collection.find({}, {:fields => ["headers", "date"]})
@final = @bs.map{|a| {:Email => a['headers']['to'], :At => a['date'], :subject => a['headers']['subject'], :Type => a['headers']['status'], :Message_id => a['headers']['message_id'] }}
render :json => @final

My data generator

 100000.times do |i|
   p i
   @coll.insert({:date =>Time.now(),:headers => {"to"=>"[email protected]", "subject"=>"meeeeeeeeee", "status" => "ffffffffffffffffff", "message_id" => "1234634673"}})
 end

Upvotes: 1

Related Questions