CODEWITHSUNDEEP
phplinuxcroncron-task
Asghar
Asghar

Reputation: 2346

Cron Jobs Managment for large amount of users

I have a service like backupify. Which Downloads data from different social media platforms, Currently i have about 2500 active users, for each user a script runs which gets data from facebook and stores them on Amazon S3, My server is Ec2 Instance on AWS.

I have entries in table like 900 entries for facebook users, There is a PHP script which runs and gets user from database table and then backups data from the facebook and then picks the next user from facebook.

Everything was fine when i was having less than 1000 users, but now i have more than 2500 users problem is that the PHP script halts, or runs for first 100 users and then halts, time out etc. I am running PHP Script fro php -q myscript.php command.

The other problem is that single user scripts takes about 65 seconds to reach the last user from the database table is may take days, so whats the best way to run parrallel on the databse table etc.

Please suggest me what is the best way to backup large amount of data for large amount of users, and i should be able to monitor the cron, somehting like a mangaer.

Upvotes: 1

Views: 461

Answers (1)

Powerslave
Powerslave

Reputation: 1428

If I get it right, you've got a single cron task for all the users, running at some frequency, trying to process the data of every user in a single shot.

  1. Did you try issuing set_time_limit(0); at the beginning of your code?
  2. Also, if the task is resource demanding, did you consider creating a separate cron task for every N user (basically mimicking multithreaded behaviour; and thus utilizing multiple CPU cores of the server)?
  3. Is writing your data to some kind of cache instead of the database and having a separate task commit the cache contents to the database feasible for you?
  4. Do you have the opportunity to use an in-memory data table (that's pretty quick)? You'll need to persist the DB contents to disk every now and then, but for this price you get a fast DB access.
  5. Can you maybe outsource the task to separate servers as a disributed service and write the cron script as a load balancer for them?
  6. Also optimizing your code might help. For example (if you're not doing so yet) you could buffer the collected data and commit in a single transaction at the end of the script so the execution flow is not scattered by DB recurring I/O blockages.

Upvotes: 2

Related Questions