CODEWITHSUNDEEP
ruby-on-railsrubydatabasestorage
user502052
user502052

Reputation: 15259

How to store (in a "smart" way) data sent periodically to my server?

I am using Ruby on Rails 3.2.13 and I have to store and treat feed data that is sent periodically to my server. I would like to store that data in a proper way since it is planned that that data will be used for building and displaying a line chart. My issue is mostly related to storing data in the database by caring system performance (since data is sent to the server each 30 seconds and can be long more than 5000 characters) and by keeping ease of data manipulation (since data includes hash-array structures).

That is, at regular intervals, my server receives data (within a HTTP request query string) that Ruby on Rails recognizes as the following:

Parameters:
{
  "feed1"=>{
    "1"=>"A,35,text1.1,5",
    "2"=>"B,76,text1.2,-6",
    "3"=>"D,45,text1.3,-1",
    "4"=>"Z,48,text1.4,-10",
    "5"=>"G,26,text1.5,-999",
    "6"=>"Y,10,text1.6,-67",
    "7"=>"T,98,text1.7,45",
    "8"=>"N,04,text1.8,315", 
    "9"=>"E,37,text1.9,90"
  },
  "feed2"=>{
    "1"=>"T,53,text2.1,76",
    "2"=>"J,67,text2.2,85",
    "3"=>"O,54,text2.3,-967",
    "4"=>"D,85,text2.4,41",
    "5"=>"N,97,text2.5,99",
    "6"=>"P,32,text2.6,48",
    "7"=>"Z,87,text2.7,21",
    "8"=>"S,27,text2.8,3", 
    "9"=>"I,67,text2.9,0"
  },
  "..."=>{...}
}

I would like to store the above data in a "smart" way and treat that data in a efficient way so to be displayed in a line chart.

How can / should I make that? There are some prescription or advice?

For example:

Upvotes: 1

Views: 175

Answers (1)

sawa
sawa

Reputation: 168189

Assuming that each feed is a simple hash, I think the format that is becoming popular for such purpose is the Labeled Tab-Separated Values (LTSV). That is, a hash:

{"a" => "foo", "b" => "bar", "c" => "baz"}

will be recorded as a single line:

a:foo\tb:bar\bc:baz\n

In your case, you can add the "time" attribute to each feed like this:

{
  "time"=>2013-07-21 02:24:01 +0900,
  "1"=>"A,35,text1.1,5",
  "2"=>"B,76,text1.2,-6",
  "3"=>"D,45,text1.3,-1",
  "4"=>"Z,48,text1.4,-10",
  "5"=>"G,26,text1.5,-999",
  "6"=>"Y,10,text1.6,-67",
  "7"=>"T,98,text1.7,45",
  "8"=>"N,04,text1.8,315", 
  "9"=>"E,37,text1.9,90"
},
...

and store that in LTSV. The code for encoding and decoding to/from LTSV is simple:

class Hash
  def to_ltsv
    map{|k, v| "#{k}:#{v.to_s.tr("\t", " ")}"}.join("\t")
  end
end

class String
  def parse_ltsv; Hash[chomp.split("\t").map{|f| f.split(":", 2)}] end
end

And since it is recorded as a text file, it is faster than using a database software.

Upvotes: 1

Related Questions