CODEWITHSUNDEEP
phpjavascriptmysqlhighcharts
glarkou
glarkou

Reputation: 7101

Enormous data and PHP errors

I am currently using the following HighCharts:HighStock:Charts: http://www.highcharts.com/stock/demo/data-grouping in order to display the data returned from the server.

We retrieve the data from a MySQL database and is really big. We are storing sensor metrics every 1 second. After a while we got the following error:

[Wed Sep 12 00:15:56 2012] [error] [client 127.0.0.1] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 4756882 bytes) in C:\\wamp\\www\\admin\\getTrends.php on line 156, referer: http://localhost/admin/trends.php
[Wed Sep 12 00:15:56 2012] [error] [client 127.0.0.1] PHP Stack trace:, referer: http://localhost/admin/trends.php
[Wed Sep 12 00:15:56 2012] [error] [client 127.0.0.1] PHP   1. {main}() C:\\wamp\\www\\admin\\getTrends.php:0, referer: http://localhost/admin/trends.php
[Wed Sep 12 00:15:56 2012] [error] [client 127.0.0.1] PHP   2. getTrendsDataAI() C:\\wamp\\www\\admin\\getTrends.php:33, referer: http://localhost/admin/trends.php
[Wed Sep 12 00:15:56 2012] [error] [client 127.0.0.1] PHP   3. printResults() C:\\wamp\\www\\admin\\getTrends.php:102, referer: http://localhost/admin/trends.php
[Wed Sep 12 00:15:56 2012] [error] [client 127.0.0.1] PHP   4. createData() C:\\wamp\\www\\admin\\getTrends.php:230, referer: http://localhost/admin/trends.php
[Wed Sep 12 00:15:56 2012] [error] [client 127.0.0.1] PHP   5. implode() C:\\wamp\\www\\admin\\getTrends.php:156, referer: http://localhost/admin/trends.php

What is the best solution to return this data as JSON object to HighStocks for viewing? And how can we overcome the PHP limitation? Shall we return chunk of data each time? How do they usually present enormous amount of data to the users and creating charts and reports from this data? Another big problem that we need to overcome is that the returned JSON object is enormous. At this point is around 20-30 mbs and it will be much larger in the future. Is it ok to return this data to the user and perform everything client side?

Any suggestions or thoughts welcome.

Upvotes: 2

Views: 322

Answers (2)

Alejandro Urbano Alvarez
Alejandro Urbano Alvarez

Reputation: 3346

When displaying large amounts of data I think the best thing you could do (And the most common one) would be to generate the views with a certain resolution.

As the user narrows down (zooms in) on a given spot, you would increase the resolution on that spot, thus reducing the overall size of the chunks

This way you could generate smaller file sizes via php, that would ideally represent the same graph. Something similar to the way google maps used to work.

Upvotes: 2

Marc B
Marc B

Reputation: 360782

JSON isn't really amendable to chunking. Each json string has to be complete in and of itself, since it directly represents a complete data structure: a string, an array, an object, etc...

What you could do is send the data over in independent chunks, and rebuild the data structure in JS. e.g. request #1 sends over 10,000 rows worth of data, request #2 gets 10,001-20,000, #3 gets 20,001-30,000 etc...

but then you're still going to end up with a huge memory footprint in your browser. neither PHP nor JS are particularly memory efficient for large structures.

Upvotes: 0

Related Questions