"undefined method []" when parsing reddit api

Question

I'm trying to request the json pages for multiple subreddits and take the title and link from each page for a college project. here's the code in question:

require 'rufus-scheduler'
require 'json'
require 'httparty'
ENV['TZ'] = 'Europe/Dublin'

scheduler = Rufus::Scheduler::singleton


scheduler.every '12h00m', :first_at => Time.now + 10 do

 array_of_subreddits = ["pics", "memes", "funny", "aww", "memes", 
"birdswitharms"]
array_of_subreddits.each do |category|
sleep 10 #wait 10 seconds between each request

@response = JSON.parse(HTTParty.get("http://reddit.com/r/#{category}/.json?limit=25").body)

  @response['data']['children'].each do |data|
      @link = data['data']['url']
      @title = data['data']['title']
      @category = category
      Pic.create([{:title => "#{@title}", :link => "#{@link}", :category => "#{@category}"}])
    end
  end
end

this works perfectly sometimes, it will run through each one and end as it should. more often than not however it will give me this message after after one or two passes:

NoMethodError (undefined method `[]' for nil:NilClass):
  app/controllers/home_controller.rb:17:in `block in index'
  app/controllers/home_controller.rb:9:in `each'
  app/controllers/home_controller.rb:9:in `index'


  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/_source.erb (4.8ms)
  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/_trace.html.erb (2.2ms)
  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/_request_and_response.html.erb (1.2ms)
  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/diagnostics.html.erb within rescues/layout (66.2ms)
  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/_markup.html.erb (0.4ms)
  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/_inner_console_markup.html.erb within layouts/inlined_string (0.3ms)
  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/_prompt_box_markup.html.erb within layouts/inlined_string (0.3ms)
  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/style.css.erb within layouts/inlined_string (0.5ms)
  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/console.js.erb within layouts/javascript (51.6ms)
  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/main.js.erb within layouts/javascript (0.3ms)
  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/error_page.js.erb within layouts/javascript (0.5ms)
  Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/index.html.erb (124.8ms)

Answer 1

This kind of errors are most common in ruby or rails. Can be handled in multiple ways. As @Stefan suggested you can use any of the bellow.

Most simply like this

response = HTTParty.get('http://reddit.com/r/#{category}/.json?limit=25')
if response.success?
  response_body = response.body
  # continue
end

or

response = HTTParty.get('http://reddit.com/r/#{category}/.json?limit=25')

case response.code
  when 200
    puts "Good!"
    # Continue your parsing
  when 404
    puts "NOT FOUND!"
  when 500...600
    puts "ERROR #{response.code}"
end

or

begin
   HTTParty.get('http://reddit.com/r/#{category}/.json?limit=25')
 rescue HTTParty::Error
   # HTTParty errors like Not found
 rescue StandardError
   # StandardError like Timeout
 else
   # continue
 end

Answer 2

Creating client classes is a much better way to work with httparty:

class RedditClient
  include HTTParty

  format :json

  base_uri "http://reddit.com/r/"

  def self.get_category(category, *opts) 
    opts.reverse_merge(limit: 25)
    get("/#{category}.json", opts)
  end
end

This way HTTParty handles JSON parsing for us and does not attempt to an convert an empty response. Its also much easier to test separately.

However you should still check if the response was successful before trying to use it:

@response = RedditClient.get_category(category)
if @response.success?
  attrs = @response['data']['children'].map do |child|
    {
       category: category,
       link: child['data']['url'],
       title: child['data']['title']
    }
  end
  Pic.create!(attrs)
else
  # log it or raise some sort of error
end

Note that you where passing an array containing a single hash to .create. You can instead pass an array of hashes and it will insert the records in a single SQL insert statement.

Answer 3

When you get errors like this, you should always dump the actual response so you can inspect it. The fact you got a error for a nil with code doing stuff like ['data']['children'] means Id guess you got a JSON response, but one that is missing one of the first items (e.g. ['data'] returned nil).

Dont just assume every request is successful, many things can make a HTTP fail. Its possible that you get a valid JSON response back, just not the one you are expecting, for example an error message which would have told you the problem.

Also even with a 10 second delay, you may be hitting a rate limit (never tested Reddit personally), but read the rules

Many default User-Agents (like "Python/urllib" or "Java") are drastically limited to encourage unique and descriptive user-agent strings.