Rails, JSON and loading times

Question

I'm having some loading time issues with my rails app and loading JSON data from a URL source which then is parsed into a graph using the lazy_high_charts gem. Currently it takes the page between 7 - 10 seconds to load each time.

I have three JSON data url's (@data, @forecast, @moisture) which are parsed using the Oj gem, as it was suggested that it would speed up the process.

I'm looking for a way to speed up this process, if at all possible.

JSON data structured like so:

{"status"=>"ok", "data"=>[{"2014-08-11 11:00:00"=>14.9},{"2014-08-11 11:30:00"=>15.1}]}

Controller

@temperature = Temperature.find(params[:id])
@temps = Temperature.find(:all, :conditions => ["id != ?", params[:id]])

@hash = Gmaps4rails.build_markers(@temperature) do |data, marker|
 marker.lat data.lat
 marker.lng data.long
end

@data =  Oj.load(open(@temperature.url).read)
@forecast =  Oj.load(open(@temperature.air_forecast).read)
@moisture =  Oj.load(open(@temperature.moisture).read)

data = []
moisture = []
forecast = []

@data['data'].flatten.each do |d|
 data << [DateTime.parse(d.keys.first).to_i * 1000, d.values.first]
end    

@moisture['data'].each do |d|
 moisture << [DateTime.parse(d.keys.first).to_i * 1000, d.values.first]
end 

@sevendays = LazyHighCharts::HighChart.new('graph') do |f|
 f.chart(:height => '400', width: '860', plotBackgroundImage: ActionController::Base.helpers.asset_path("chartbg.png"))
 f.yAxis [:title => {:text => "Soil Temperature (\u00B0C)", :margin => 20, style: { color: '#333'}}, min: 0, plotLines: [{ color: '#b20838', width: 2, value: 28 }], ]
 f.series(:type => 'line', :name => 'Soil Temperature', data: data, marker: {enabled: false}, :color => '#00463f' )
 f.xAxis(:type => 'datetime', tickInterval: 1.day.to_i * 1000, :dateTimeLabelFormats => { day: "%b %e"}, :tickmarkPlacement => 'on', :startOnTick => true, min: 1.weeks.ago.at_midnight.to_i * 1000, labels: { y: 20 } )
 f.legend({:align => 'center', :verticalAlign => 'top', :y => 0, :borderWidth => 0, style: {color: "#333"}})
end

@day = LazyHighCharts::HighChart.new('graph') do |f|
 f.chart(:height => '400', width: '860', plotBackgroundImage: ActionController::Base.helpers.asset_path("chartbg.png"))
 f.yAxis [:title => {:text => "Soil Temperature (\u00B0C)", :margin => 20, style: { color: '#333'}}, min: 0, plotLines: [{ color: '#b20838', width: 2, value: 28 }]]
 f.series(:type => 'line', :name => 'Soil Temperature', data: data, marker: {enabled: false}, :color => '#00463f' )
 f.xAxis(:type => 'datetime', tickInterval: 5.hour.to_i * 1000, :dateTimeLabelFormats => { day: "%b %e"}, :tickmarkPlacement => 'on', :startOnTick => true, min: 1.day.ago.at_midnight.to_i * 1000, labels: { y: 20 } )
 f.legend({:align => 'center', :verticalAlign => 'top', :y => 0, :borderWidth => 0, style: {color: "#333"}})
end

@month = LazyHighCharts::HighChart.new('graph') do |f|
 f.chart(:height => '400', width: '860', plotBackgroundImage: ActionController::Base.helpers.asset_path("chartbg.png"))
 f.yAxis [:title => {:text => "Soil Temperature (\u00B0C)", :margin => 20, style: { color: '#333'}}, min: 0, plotLines: [{ color: '#b20838', width: 2, value: 28 }]]
 f.series(:type => 'line', :name => 'Soil Temperature', data: data, marker: {enabled: false}, :color => '#00463f' )
 f.xAxis(:type => 'datetime', tickInterval: 2.day.to_i * 1000, :dateTimeLabelFormats => { day: "%b %e"}, :tickmarkPlacement => 'on', :startOnTick => true, min: 1.month.ago.at_midnight.to_i * 1000, labels: { rotation: 90, y:20 })
 f.legend({:align => 'center', :verticalAlign => 'top', :y => 0, :borderWidth => 0, style: {color: "#333"}})
 f.plotOptions({line: {turboThreshold: 1500}})
end

@moisture_graph = LazyHighCharts::HighChart.new('graph') do |f|
 f.chart(:height => '400', width: '860', plotBackgroundImage: ActionController::Base.helpers.asset_path("chartbg.png"))
 f.yAxis [:title => {:text => "Litres of Water Per Litre of Soil", :margin => 20, style: { color: '#333'}}]
 f.series(:type => 'line', :name => 'Surface Moisture Volume', pointInterval: 1.day * 1000, data: moisture, marker: {enabled: false}, :color => '#00463f' )
 f.xAxis(:type => 'datetime', tickInterval: 1.day.to_i * 1000, :dateTimeLabelFormats => { day: "%b %e"}, :tickmarkPlacement => 'on', :startOnTick => true, min: 1.weeks.ago.at_midnight.to_i * 1000, labels: { y: 20 } )
 f.legend({:align => 'center', :verticalAlign => 'top', :y => 0, :borderWidth => 0, style: {color: "#333"}})
end

respond_to do |format|
 format.html # show.html.erb
 format.json { render json: @temperature }
end
end

View

<ul class="nav nav-pills">
 <li class="active"><a href="#seven" data-toggle="tab">Last 7 Days</a></li>
 <li><a href="#forecast" data-toggle="tab">7 Day Forecast Temperature</a></li>
 <li><a href="#day" data-toggle="tab">Last 24 Hours</a></li>
 <li><a href="#month" data-toggle="tab">Last 30 Days</a></li>
 <li><a href="#moisture" data-toggle="tab">Surface Moisture Volume</a></li>
 <li><a href="#location" data-toggle="tab">Location of <%= @soil_temperature.property %></a></li>
</ul>

<div class="tab-content">

    <div class="tab-pane active" id="seven" style="width: 100%;">
        <%= high_chart("chart", @sevendays) %>
    </div>

    <div class="tab-pane" id="forecast" style="width: 100%;">
        <table class="table table-bordered table_forecast" width="100%">
            <th></th>
            <% Time.use_zone('Sydney'){(0.day.from_now.to_date..6.day.from_now.to_date)}.each do |d| %>
            <th><%= d.strftime("%a %d, %b") %></th>
            <% end %>
            <tr>
            <td width="12.5%"><b>Maximum</b></td>
            <% Time.use_zone('Sydney'){(0.day.from_now.to_date..6.day.from_now.to_date)}.each do |d| %>
            <% if @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.max { |a,b| a[1] <=> b[1] }[1] > 20 %>
            <td width="12.5%" bgcolor="#ed1c24"><font color="#ffffff">
            <% elsif @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.max { |a,b| a[1] <=> b[1] }[1] > 14 %>
            <td width="12.5%" bgcolor="#f58233"><font color="#fff">
            <% elsif @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.min { |a,b| a[1] <=> b[1] }[1] < 14 %>
            <td width="12.5%" bgcolor="#00a1e4"><font color="#fff">
            <% else %>
            <td width="12.5%"><font color="#333333">
            <% end %>
            <%= @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.max { |a,b| a[1] <=> b[1] }[1] %>°C</font></td>
            <% end %>
            </tr>
            <tr>
            <td width="12.5%"><b>Minimum</b></td>
            <% Time.use_zone('Sydney'){(0.day.from_now.to_date..6.day.from_now.to_date)}.each do |d| %>
            <% if @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.min { |a,b| a[1] <=> b[1] }[1] > 20 %>
            <td width="12.5%" bgcolor="#ed1c24"><font color="#ffffff">
            <% elsif @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.min { |a,b| a[1] <=> b[1] }[1] > 14 %>
            <td width="12.5%" bgcolor="#f58233"><font color="#fff">
            <% elsif @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.min { |a,b| a[1] <=> b[1] }[1] < 14 %>
            <td width="12.5%" bgcolor="#00a1e4"><font color="#fff">
            <% else %>
            <td width="12.5%"><font color="#333333">
            <% end %>
            <%= @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.min { |a,b| a[1] <=> b[1] }[1] %>°C</td>
            <% end %>
            </tr>
            <tr>
            <td width="12.5%"><b>Average</b></td>
            <% Time.use_zone('Sydney'){(0.day.from_now.to_date..6.day.from_now.to_date)}.each do |d| %>
            <% if (@forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.sum { |sum| sum[1] } / @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.count) > 20 %>
            <td width="12.5%" bgcolor="#ed1c24"><font color="#ffffff">
            <% elsif (@forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.sum { |sum| sum[1] } / @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.count) > 14 %>
            <td width="12.5%" bgcolor="#f58233"><font color="#fff">
            <% elsif (@forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.sum { |sum| sum[1] } / @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.count) < 14 %>
            <td width="12.5%" bgcolor="#00a1e4"><font color="#fff">
            <% else %>
            <td width="12.5%"><font color="#333333">
            <% end %>
            <%= "%.1f" % (@forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.sum { |sum| sum[1] } / @forecast['data']['temperatures'].select { |temp| temp[0].to_date == d }.count) %>°C</td>
            <% end %>
            </tr>

        </table>
    </div>

    <div class="tab-pane" id="day" style="width: 100%;">
        <%= high_chart("chart2", @day) %>
    </div>

    <div class="tab-pane" id="month" style="width: 100%;">
        <%= high_chart("chart3", @month) %>
    </div>

    <div class="tab-pane" id="moisture" style="width: 100%;">
        <%= high_chart("chart4", @moisture_graph) %>
    </div>

    <div class="tab-pane" id="location" style="width: 100%;">
        <div id="map" style='width: 100%; height: 600px;'></div>
    </div>
    <hr>

</div>

<script type="text/javascript">

    $('a[href="#location"]').on('shown', function(e) {
    var mapOptions = { mapTypeId: google.maps.MapTypeId.HYBRID, Zoom: 9 };
    handler = Gmaps.build('Google');
    handler.buildMap({ provider: mapOptions, internal: {id: 'map'}}, function(){
    markers = handler.addMarkers(<%= raw @hash.to_json %>);
    handler.map.centerOn(markers[0]); 
    handler.getMap().setZoom(9);
    google.maps.event.trigger(map, "resize");
    });
  });

</script>

Answer 1

1. request data in parallel

@romainsalles is right, if fetching the data is a part of what's slow then your best bet is to parallelize that. I don't have much experience with Celluloid. With threads you can do it like this:

t1 = Thread.new{ @data     =  Oj.load(open(@temperature.url).read) }
t2 = Thread.new{ @forecast =  Oj.load(open(@temperature.air_forecast).read) }
t3 = Thread.new{ @moisture =  Oj.load(open(@temperature.moisture).read) }

t1.join
t2.join
t3.join

2. request gziped data

If the place you are requesting your data from supports compressing responses, then you can switch to an http client which accepts gziped responses. This might reduce transmission time by anywhere from 2x-10x depending on the data.

I believe httpclient will do this by default.

require 'httpclient'
clnt = HTTPClient.new
clnt.get_content(url) #instead of open(url).read

Answer 2

It sounds as if the "URL source" is giving weather forecast information from a third-party site. Do the URLs themselves take 7-10 seconds to load? If so, there's probably very little you can do to speed them up. Even parallel requests will only be as fast as the fastest request. Instead, you can try to move the delay so the end user doesn't experience it.

Presumably, this information doesn't change significantly minute-to-minute. If you have a limited number of potential forecast locations, you could try loading and storing the data beforehand. In particular, you could use a cron job to pull the data every X minutes and store it in your database, where you can query it on page load.

Alternatively, you can load the page without the data (showing a simple "Loading..." text) and then populate it with the data via Ajax. This wouldn't make it any faster to view the data, but it would provide a better user experience than staring at a blank page for 7-10 seconds.