CODEWITHSUNDEEP
rubyubuntunokogiri
B A
B A

Reputation: 1119

Parsing huge XML in Ruby doesn't process full file in server

I need to parse a 3MB XML file using Nokogiri and Httparty to parse content from the links in the XML.

It works fine when I do it on my Windows 7 PC but when I run the same script in a VPS, it doesn't seem to process it fully.

Is it because of the limit on the maximum execution time of a Ruby script on the server? If so, how can I increase that in Ubuntu?

Upvotes: 0

Views: 76

Answers (1)

djaszczurowski
djaszczurowski

Reputation: 4523

Noone will tell you exactly why it happens without logs. Did you browse them?

As far as VPS and xml are concerned - you may exceed the maximum available memory on the server (what will happen quickly if you have multiple clients, all of them loading xmls). I'm not sure how you are using Nokogiri, but my advise is to check whether you're using SAX parser (http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/SAX/Parser)

Eventually you will use less resources (memory, execution time) but there is a drawback - SAX parsers are much harder to write, especially when xml structure is complicated, but it may be wort (look for benchmarks "Nokogiri DOM vs SAX")

Upvotes: 2

Related Questions