CODEWITHSUNDEEP
javajsonapache-commons-httpclient
jorne
jorne

Reputation: 903

parsing giant json response

First the background info: I'm using commons httpclient to make a get request to the server. Server response a json string. I parse the string using org.json.

The problem: Actually everything works, that is for small responses (smaller then 2^31 bytes = max value of an integer which limits the getResponseBody and the stringbuilder). I on the other hand have a giant response (over several GB) and I'm getting stuck. I tried using the "getResponseBodyAsStream" of httpclient but the response is so big that my system is getting stuck. I tried using a String, a Stringbuilder, even saving it to a file.

The question: First, is this the right approach, if so, what is the best way to handle such a response? If not, how should I proceed?

Upvotes: 1

Views: 224

Answers (1)

vach
vach

Reputation: 11387

If you ever shall have a response which can be factor of GB you shall parse the json as stream character by character (almost) and avoid creating any String objects... (its very important because java stopTheWorld garbage collection will cause your system freese for seconds if you constantly create lot of garbage)

You can use SAXophone to create parsing logic.

You'll have to implement all the methods like onObjectStart, onObjectClose, onObjectKey etc... its hard at first but once you take a look implementation of PrettyPrinter in test packages you'll have the idea...

Once properly implemented you can handle an infinite stream of data ;)

P.S. this is designed for HFT so its all about performance and no garbage...

Upvotes: 2

Related Questions