How to improve on PHP's XML loading time?

Question

Dropping my lurker status to finally ask a question...

I need to know how I can improve on the performance of a PHP script that draws its data from XML files.

Some background:

• I've already mapped the bottleneck to CPU - but want to optimize the script's performance before taking a hit on processor costs. Specifically, the most CPU-consuming part of the script is the XML loading.

• The reason I'm using XML to store object data because the data needs to be accessible via a browser Flash interface, and we want to provide fast user access in that area. The project is still in early stages though, so if best practice would be to abandon XML altogether, that would be a good answer too.

• Lots of data: Currently plotting for roughly 100k objects, albeit usually small ones - and they must ALL be taken up into the script, with perhaps a few rare exceptions. The data set will only grow with time.

• Frequent runs: Ideally, we'd run the script ~50k times an hour; realistically, we'd settle for ~1k/h runs. This coupled with data size makes performance optimization completely imperative.

• Already taken an optimization step of making several runs on the same data rather than loading it for each run, but it's still taking too long. The runs should generally use "fresh" data with the modifications done by users.

Answer 1

~1k/hour, 3600 seconds per hour, more than 3 runs a second (let alone the 50k/hour)...

There are many questions. Some of them are:

• Does your php script need to read/process all records of the data source for each single run? If not, what kind of subset does it need (~size, criterias, ...)
• Same question for the flash application + who's sending the data? The php script? "Direct" request for the complete, static xml file?
• What operations are performed on the data source?
• Do you need some kind of concurrency mechanism?
• ...

And just because you want to deliver xml data to the flash clients it doesn't necessarily mean that you have to store xml data on the server. If e.g. the clients only need a tiny little subset of the availabe records it probably a lot faster not to store the data as xml but something more suited to speed and "searchability" and then create the xml output of the subset on-the-fly, maybe assisted by some caching depending on what data the client request and how/how much the data changes.

edit: Let's assume that you really,really need the whole dataset and need a continuous simulation. Then you might want to consider a continuous process that keeps the complete "world model" in memory and operates on this model on each run (world tick). This way at least you wouldn't have to load the data on each tick. But such a process is usually written in something else than php.

Answer 2

Just to clarify: is the data you're loading coming from XML files for processing in its current state and is it being modified before being sent to the Flash application?

It looks like you'd be better off using a database to store your data and pushing out XML as needed rather than reading it in XML first; if building the XML files gets slow you could cache files as they're generated in order to avoid redundant generation of the same file.

Answer 3

If the XML stays relatively static, you could cache it as a PHP array, something like this:

<xml><foo>bar</foo></xml>

is cached in a file as

<?php return array('foo' => 'bar');

It should be faster for PHP to just include the arrayified version of the XML.