CODEWITHSUNDEEP
phpbashcron
MagnusD
MagnusD

Reputation: 35

Having Cron job execute script in background and run for 15 hours

I have a PHP script that takes about 15 hours to run and I have to run it once a week. The script downloads and extracts the contents of a massive (70gb) CSV file into our database.

I need the script to start running every Sunday at 6 PM and to keep running until it's finished processing the file.

My plan is to create a bash file cron.sh that has the following:

nohup php /var/www/html/cron.php > /var/www/html/outputs/`date "+%Y-%m-%d %H:%M:%S"`.log

And the cron being:

0 18 * * 0 /var/www/html/cron.sh

How would I be able to improve this?

Upvotes: 2

Views: 3186

Answers (2)

Zak
Zak

Reputation: 7525

Since your OP asked "How would I be able to improve this" -- You didn't mention if you HAD to use PHP.

If the CSV is that large .. I would create a bash file that does the same thing that your PHP file does (since you're already calling cron.sh). You would have to change your PHP configuration to less than ideal settings to be able to run it efficiently. IMHO straight bash is the way to go. That way nohup can be bypassed entirely.

Meaning I would:

1) Create a login path for MySQL to securely connect to your db from the .sh file. You can acheive this with the following command:

sudo mysql_config_editor set --login-path=client --host=yourHost --user=MySQL_user --password

You will be prompted for the MySQL password.

2) Once that is done, you can access MySQL in your bash file like such AND create your logs all in the same file:

#!/bin/bash

cd /place/where/you/want/the/csv/

wget http://your_file_location.com/csv/location/file.csv
# Or whatever you're doing to "download" it -- Just an example .. 

mysql --login-path=local infile=1 3parsfdb -e "LOAD DATA LOCAL INFILE 'file.csv' INTO TABLE your_table FIELDS TERMINATED BY ','"

echo "Whatever you've collected" >> /var/www/html/outputs/`date "+%Y-%m-%d %H:%M:%S"`.log

Of course that's just an example .. But you get the general idea. Then store said bash file in a safe location.

3) Put your newly created bash file into the crontab. You can use your user's crontab, or if the user doesn't have access to the bash file or directory you put it in, you can use root's crontab -- IE:

0 18 * * 0 /path/to/your/bash/file/BackupCSVToDB.sh

The saved overhead of not having to use PHP could save the run-time literally hours.

Upvotes: 2

kontiki
kontiki

Reputation: 40639

Remember to redirect stderr as well:

nohup php /var/www/html/cron.php > dir/stdout.log 2> dir/stderr.log

or if you prefer to merge them all in the same file:

nohup php /var/www/html/cron.php > dir/stdout.log 2>&1

The 2>&1 means redirect stderr to the same file stdout is using.

The rest is all fine, putting the command inside a shell script makes it more readable if the line is too long. Also if you later want to make the task do more things, it is easier to update the script file.

I would probably rename the cron.sh file to something more meaningful.

Upvotes: 2

Related Questions