CODEWITHSUNDEEP
phpxmldatabasemultithreadingimport
olyar
olyar

Reputation: 64

Big XML file import to database PHP

I am facing a problem that somehow I don't see the solution to it. I have a XML file that needs to be importet to custom DB structure, when user uploads / imports the file the ajax post is waiting untill the file import is finished, but this could take 5 hours or more I don't know. What is the best way to handle this UI issue.

Upvotes: 0

Views: 488

Answers (1)

ArtisticPhoenix
ArtisticPhoenix

Reputation: 21661

You would want to run them using a background job ( a detached process ) this way the end user gets a confirmation message right away, and then send an email when the long running task is complete. Then they don't have to wait for it to finish. As I mentioned in the comments I have a class I wrote on my git hub for this

https://github.com/ArtisticPhoenix/MISC/blob/master/BgProcess.php

But it passes the args as a path because it's setup for Code Igniter, so you would have to change that or split the arguments up within your code.

Anyway the basics is similar to running a cron job, This varies in the implantation depending on the OS of the server. But on Linux the command is like this

 php -f "path/to/phpfile.php" "{args}" > /dev/null &

The > /dev/null & part sends the output to null ( throws it away ) and the & runs it as a non-blocking process meaning the script starting the command can continue on. So using an example as this

 .. other code before starting background job ..
 exec( 'php -f "path/to/phpfile/xmlProcessor.php" "testXML/2" > /dev/null &');
  .. code to tell user job is started .. this runs right after the call without waiting for that process to finish.

Then in xmlProcessor.php you would have this

  <?php

         $args = explode('/', $argv[1]);
         $file = $ags[0];
         $user_id = $args[1];
         ... code to process xml 
          ... email user confirmation of completion

http://php.net/manual/en/reserved.variables.argv.php

As I said typically would call it this way, 

  exec( 'php -f "path/to/phpfile/xmlProcessor.php" "testXML" "2" > /dev/null &');

And access them using 

   $argv[1] // = testXML
   $argv[2] // = 2

But because I use this with CI, it does it's routing for me to a special controller and handles all that. The nice thing about my class is that it should find the PHP executable under most cases, and it has windows compatibility built in ( which was a pain in the ...)

Using that class you would just call it like this

 $command = new BgProcess( "path/to/phpfile/xmlProcessor.php", "testXML", 2);

 echo $command;

Would output 'php -f "path/to/phpfile/xmlProcessor.php" "testXML/2" > /dev/null &' after starting the process ( the return is just for debugging )

Basically your running a separate background job with PHP via the command line.

Upvotes: 1

Related Questions