CODEWITHSUNDEEP
phpjsonlarge-files
The Mighty Rubber Duck
The Mighty Rubber Duck

Reputation: 4478

Processing large JSON files in PHP

I am trying to process somewhat large (possibly up to 200M) JSON files. The structure of the file is basically an array of objects.

So something along the lines of:

[
  {"property":"value", "property2":"value2"},
  {"prop":"val"},
  ...
  {"foo":"bar"}
]

Each object has arbitrary properties and does not necessary share them with other objects in the array (as in, having the same).

I want to apply a processing on each object in the array and as the file is potentially huge, I cannot slurp the whole file content in memory, decoding the JSON and iterating over the PHP array.

So ideally I would like to read the file, fetch enough info for each object and process it. A SAX-type approach would be OK if there was a similar library available for JSON.

Any suggestion on how to deal with this problem best?

Upvotes: 29

Views: 39904

Answers (7)

Nigel Ren
Nigel Ren

Reputation: 57121

I know that the JSON streaming parser https://github.com/salsify/jsonstreamingparser has already been mentioned. But as I have recently(ish) added a new listener to it to try and make it easier to use out of the box I thought I would (for a change) put some information out about what it does...

There is a very good write up about the basic parser at https://www.salsify.com/blog/engineering/json-streaming-parser-for-php, but the issue I have with the standard setup was that you always had to write a listener to process a file. This is not always a simple task and can also take a certain amount of maintenance if/when the JSON changed. So I wrote the RegexListener.

The basic principle is to allow you to say what elements you are interested in (via a regex expression) and give it a callback to say what to do when it finds the data. Whilst reading the JSON, it keeps track of the path to each component - similar to a directory structure. So /name/forename or for arrays /items/item/2/partid- this is what the regex matches against.

An example is (from the source on github)...

$filename = __DIR__.'/../tests/data/example.json';
$listener = new RegexListener([
    '/1/name' => function ($data): void {
        echo PHP_EOL."Extract the second 'name' element...".PHP_EOL;
        echo '/1/name='.print_r($data, true).PHP_EOL;
    },
    '(/\d*)' => function ($data, $path): void {
        echo PHP_EOL."Extract each base element and print 'name'...".PHP_EOL;
        echo $path.'='.$data['name'].PHP_EOL;
    },
    '(/.*/nested array)' => function ($data, $path): void {
        echo PHP_EOL."Extract 'nested array' element...".PHP_EOL;
        echo $path.'='.print_r($data, true).PHP_EOL;
    },
]);
$parser = new Parser(fopen($filename, 'r'), $listener);
$parser->parse();

Just a couple of explanations...

'/1/name' => function ($data)

So the /1 is the the second element in an array (0 based), so this allows accessing particular instances of elements. /name is the name element. The value is then passed to the closure as $data

"(/\d*)" => function ($data, $path )

This will select each element of an array and pass it one at a time, as it's using a capture group, this information will be passed as $path. This means when a set of records is present in a file, you can process each item one at a time. And also know which element without having to keep track.

The last one

'(/.*/nested array)' => function ($data, $path):

effectively scans for any elements called nested array and passes each one along with where it is in the document.

Another useful feature I found was that if in a large JSON file, you just wanted the summary details at the top, you can grab those bits and then just stop...

$filename = __DIR__.'/../tests/data/ratherBig.json';
$listener = new RegexListener();
$parser = new Parser(fopen($filename, 'rb'), $listener);
$listener->setMatch(["/total_rows" => function ($data ) use ($parser) {
    echo "/total_rows=".$data.PHP_EOL;
    $parser->stop();
}]);

This saves time when you are not interested in the remaining content.

One thing to note is that these will react to the content, so that each one is triggered when the end of the matching content is found and may be in various orders. But also that the parser only keeps track of the content you are interested in and discards anything else.

If you find any interesting features (sometimes horribly know as bugs), please let me know or report an issue on the github page.

Upvotes: 0

Filip Halaxa
Filip Halaxa

Reputation: 778

Recently I made a library called JSON Machine, which efficiently parses unpredictably big JSON files. Usage is via simple foreach. I use it myself for my project.

Example:

foreach (JsonMachine::fromFile('employees.json') as $employee) {
    $employee['name']; // etc
}

See https://github.com/halaxa/json-machine

Upvotes: 6

user3942918
user3942918

Reputation: 26375

I've written a streaming JSON pull parser pcrov/JsonReader for PHP 7 with an api based on XMLReader.

It differs significantly from event-based parsers in that instead of setting up callbacks and letting the parser do its thing, you call methods on the parser to move along or retrieve data as desired. Found your desired bits and want to stop parsing? Then stop parsing (and call close() because it's the nice thing to do.)

(For a slightly longer overview of pull vs event-based parsers see XML reader models: SAX versus XML pull parser.)

Example 1:

Read each object as a whole from your JSON.

use pcrov\JsonReader\JsonReader;

$reader = new JsonReader();
$reader->open("data.json");

$reader->read(); // Outer array.
$depth = $reader->depth(); // Check in a moment to break when the array is done.
$reader->read(); // Step to the first object.
do {
    print_r($reader->value()); // Do your thing.
} while ($reader->next() && $reader->depth() > $depth); // Read each sibling.

$reader->close();

Output:

Array
(
    [property] => value
    [property2] => value2
)
Array
(
    [prop] => val
)
Array
(
    [foo] => bar
)

Objects get returned as stringly-keyed arrays due (in part) to edge cases where valid JSON would produce property names that are not allowed in PHP objects. Working around these conflicts isn't worthwhile as an anemic stdClass object brings no value over a simple array anyway.

Example 2:

Read each named element individually.

$reader = new pcrov\JsonReader\JsonReader();
$reader->open("data.json");

while ($reader->read()) {
    $name = $reader->name();
    if ($name !== null) {
        echo "$name: {$reader->value()}\n";
    }
}

$reader->close();

Output:

property: value
property2: value2
prop: val
foo: bar

Example 3:

Read each property of a given name. Bonus: read from a string instead of a URI, plus get data from properties with duplicate names in the same object (which is allowed in JSON, how fun.)

$json = <<<'JSON'
[
    {"property":"value", "property2":"value2"},
    {"foo":"foo", "foo":"bar"},
    {"prop":"val"},
    {"foo":"baz"},
    {"foo":"quux"}
]
JSON;

$reader = new pcrov\JsonReader\JsonReader();
$reader->json($json);

while ($reader->read("foo")) {
    echo "{$reader->name()}: {$reader->value()}\n";
}

$reader->close();

Output:

foo: foo
foo: bar
foo: baz
foo: quux

How exactly to best read through your JSON depends on its structure and what you want to do with it. These examples should give you a place to start.

Upvotes: 23

Aaron Averill
Aaron Averill

Reputation: 233

This is a simple, streaming parser for processing large JSON documents. Use it for parsing very large JSON documents to avoid loading the entire thing into memory, which is how just about every other JSON parser for PHP works.

https://github.com/salsify/jsonstreamingparser

Upvotes: 2

The Mighty Rubber Duck
The Mighty Rubber Duck

Reputation: 4478

I decided on working on an event based parser. It's not quite done yet and will edit the question with a link to my work when I roll out a satisfying version.

EDIT:

I finally worked out a version of the parser that I am satisfied with. It's available on GitHub:

https://github.com/kuma-giyomu/JSONParser

There's probably room for some improvement and am welcoming feedback.

Upvotes: 16

joni
joni

Reputation: 5462

There exists something like this, but only for C++ and Java. Unless you can access one of these libraries from PHP, there's no implementation for this in PHP but json_read() as far as I know. However, if the json is structured that simple, it's easy to just read the file until the next } and then process the JSON received via json_read(). But you should better do that buffered, like reading 10kb, split by }, if not found, read another 10k, and else process the found values. Then read the next block and so on..

Upvotes: 2

Alex Jasmin
Alex Jasmin

Reputation: 39496

There is http://github.com/sfalvo/php-yajl/ I didn't use it myself.

Upvotes: 0

Related Questions