Andrew Joll
Reputation: 31
ElasticBeanstalk worker cron running multiple times per minute
I've got an EB worker environment (single instance) running Laravel which is used to run both scheduled and dispatched jobs.
The problem I'm facing is that cron is being called multiple times per minute, leading to scheduled jobs doubling up. This can be seen in the worker's access log;
-------------------------------------
/var/log/httpd/access_log
-------------------------------------
127.0.0.1 (-) - - [27/Feb/2019:00:07:01 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:01 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:01 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:01 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:01 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:07 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:07 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:08 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:09 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:09 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:09 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:10 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:11 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:07:11 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:15 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:16 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:08:48 +0000] "POST /scheduled.php HTTP/1.1" 302 215 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:09:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:09:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:09:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:09:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:09:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:09:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:09:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:09:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:09:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
127.0.0.1 (-) - - [27/Feb/2019:00:09:00 +0000] "POST /worker/schedule HTTP/1.1" 302 217 "-" "aws-sqsd/2.4"
Leading to similar duplication in my CloudWatch logs;
20:40:00 INFO: Processing /2019/20190225.csv [] []
20:40:00 INFO: Processing /2019/20190225.csv [] []
20:40:00 INFO: Processing /2019/20190225.csv [] []
20:40:01 INFO: Processing /2019/20190225.csv [] []
20:40:01 INFO: Processing /2019/20190225.csv [] []
20:40:02 INFO: Processing /2019/20190225.csv [] []
20:40:02 INFO: Processing /2019/20190225.csv [] []
20:40:03 INFO: Processing /2019/20190225.csv [] []
20:40:03 INFO: Processing /2019/20190225.csv [] []
20:40:04 INFO: Processing /2019/20190225.csv [] []
20:40:04 INFO: Processing /2019/20190225.csv [] []
20:40:06 INFO: Processing /2019/20190225.csv [] []
20:40:06 INFO: Processing /2019/20190225.csv [] []
20:40:06 INFO: Processing /2019/20190225.csv [] []
20:40:06 INFO: Processing /2019/20190225.csv [] []
20:40:06 INFO: Processing /2019/20190225.csv [] []
20:40:06 INFO: Processing /2019/20190225.csv [] []
I'm using dusterio/laravel-aws-worker to interact with an SQS queue to process dispatched jobs.
The worker has the following in cron.yaml;
version: 1
cron:
- name: "schedule"
url: "/worker/schedule"
schedule: "* * * * *"
My worker's console kernel is scheduling an hourly job like this;
protected function schedule(Schedule $schedule)
{
$schedule->job(new ImportData())
->hourly()
->timezone(config('app.timezone')); // Pacific/Auckland
}
Is there something I'm missing here?
Upvotes: 1
Views: 548
Answers (1)
Andrew Joll
Reputation: 31
For those who stumble across this, the problem was that HTTPS redirection was happening on my app, and the same app is being deployed to both the web tier and the worker tier. The 302 response would cause SQS to retry up to your max retries limit, but each call was still executing the job.
One solution is to exclude the SQS agent from this redirection;
files:
/etc/httpd/conf.d/http-redirect.conf:
mode: "000644"
owner: root
group: root
content: |
RewriteEngine On
RewriteCond %{HTTP:X-Forwarded-Proto} !https
RewriteCond %{HTTP_USER_AGENT} !ELB-HealthChecker
RewriteCond %{HTTP_USER_AGENT} !aws-sqsd
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
Another way would be to look at preventing this script from being deployed to the worker environment.
Upvotes: 2
Related Questions
- How do I setup and use Laravel Scheduling on AWS Elastic Beanstalk?
- Amazon Elastic Beanstalk Worker cronjob (SQS) triggers same message multiple times
- AWS Elastic Beanstalk, running a cronjob
- Cron job not working in Elastic Beanstalk
- Multiple crons in AWS Elastic Beanstalk
- elastic beanstalk cron run twice
- Aws Elasticbeanstalk cron.yaml worker issue
- Single instance cronjob in Amazon ElasticBeanstalk
- AWS Elastic Beanstalk CRON jobs setup
- Double execution (2 threads) of Spring cron job in AWS Beanstalk