Yajl to stream huge json file with array data and act on each record

Question

I have a huge json file with Array-data containing json records from a db. Lets say below

[
{"id":2,"name":"def","price":3.714714089426208},
{"id":3,"name":"ghi","price":27.0179624376119},
{"id":1,"name":"abc","price":52.51248221170203}
]

I would like to stream the file and process one record at a time. Please advise how this is done.

I tried below code. But it prints whole array, my concern is memory usage I would like to process one record at a time so there is no need to load whole array into memory.

class JsonStreamer

  attr_accessor :parser

  def initialize()
    post_init
  end

  def post_init
    @parser = Yajl::Parser.new(:symbolize_keys => true)
    @parser.on_parse_complete = lambda {|obj|
      puts obj
      puts "-----------------------End of object --------------------------------"
    }
  end

  def receive_data(data)
    # continue passing chunks
    @parser << data
  end

  def parse(f)
    File.open(f, 'r') do |f|
      f.each(100) {|chunk| receive_data (chunk)}
    end

  end

end

js = JsonStreamer.new()
js.parse('sample.json')

Answer 1

I suggest checking out the json-streamer gem's example.