CODEWITHSUNDEEP
phpbackgroundworkeramazon-elastic-beanstalksupervisordbeanstalkd
Tariq Albajjali
Tariq Albajjali

Reputation: 337

Running beanstalk workers hangs the server

I have five beanstalk workers written in PHP, each one is watching a certain tube and process the data, sending emails mainly, messages emails, notifications emails and signup emails, each worker are talking to the database and fetching some data before sending the emails.

I tried running the workers using "Screen" and "crontab" and i am trying "supervisord" at the moment, but whatever i am using, the server hangs and it gives me "memory allocation error" so i have to reboot the server each time, my local server is 8GB Memory, what should i put under consideration when running the workers, memory for example, CPU usage, can anyone point me where to look ?

here is a sample worker for sending messages emails

public function actionMessages() {
    $pheanstalk = new Pheanstalk("127.0.0.1");

    while (true) {
        // worker (performs jobs)
        $job = $pheanstalk
                ->watch('MessageEmail')
                ->ignore('default')
                ->reserve();

        $jobData = json_decode($job->getData(), true);

        $usersObj = new Users();
        $circlesObj = new Circles();
        $messageId = NULL;
        $listUsers = [];


        $finalDataArray = [];
        $finalDataUsersArray = [];
        foreach ($jobData['circlesIds'] as $circle) {
            // get users in lists 

            foreach ($jobData['lists'] as $list) {
                $listsObj = new Lists();
                $listUsers = $listsObj->getListInfoWithOutsiders($list);
            }

            $circleInfo = $circlesObj->getBasicInfoById($circle);

            if (sizeof($listUsers) > 0) {

                // send emails to the list 
                foreach ($listUsers as $userInListId) {
                    // check if the user is circlu or outsider
                    if ($userInListId['outsider']) {
                        $circlesProfilesObj = new CirclesMembersManagement();
                        $userInfo = $circlesProfilesObj->getMemberEmailById($userInListId['user_id']);


                        $finalDataArray[] = ['circle' => $circleInfo, 'email' => $userInfo['email'],
                            'post' => ['html' => $jobData['messageText'], 'date' => $jobData['postDate']]];

                        // send email 
                    } else {
                        $userInfo = $usersObj->getUserEmailById($userInListId['user_id']);

                        // insert the main message 
                        $messages = new Messages();
                        $messageId = $messages->addCircleMessage($jobData['messageText'], $userInListId['user_id'], $circleInfo['id']);

                        $finalDataArray[] = ['circle' => $circleInfo, 'email' => $userInfo['email'],
                            'post' => ['html' => $jobData['messageText'], 'date' => $jobData['postDate']]];
                    }
                }

                EmailHelper::sendMessageEmail($finalDataArray);
            }

            // send messages to the the users 
            if (sizeof($jobData['users']) > 0) {

                foreach ($jobData['users'] as $userId) {

                    // insert the main message 
                    if ($messageId) {
                        // send direct message 
                        $userMessagesObj = new UserMessages();
                        $userMessagesObj->sendCircleMessage($circleInfo['id'], $userId, $messageId);
                    } else {
                        $messages = new Messages();
                        $messages->addCircleMessage($jobData['messageText'], $userId, $circleInfo['id']);
                    }

                    $userInfo = $usersObj->getUserEmailById($userId);

                    $finalDataUsersArray[] = ['circle' => $circleInfo, 'email' => $userInfo['email'],
                        'post' => ['html' => $jobData['messageText'], 'date' => $jobData['postDate']]];

                    // send email 
                }
                EmailHelper::sendMessageEmail($finalDataUsersArray);
            }
        }

        $pheanstalk->delete($job);
    }
}

Thanks in advance

Upvotes: 0

Views: 384

Answers (1)

Pentium10
Pentium10

Reputation: 208002

So If you are running out of memory, you are leaking somewhere memory. Please investigate this how to find and locate it.

Otherwise, it's an acceptable technique to stop and restart the worker once after a batch of workers or time. Eg: after every 10k message restart the worker, or every 24hrs restart the worker. This would close the handlers, free memory and start fresh. How to achieve this is another interesting topic, it could be self managed, like exit. Than the supervisor would relaunched it.

The best would be to place all these in threads. And each job would be handled in it's own child thread, this is how we run workers now in PHP and we run like this to avoid memory leaking and handlers.

Also it's good to setup a broker or an ochestrator which can be a simple PHP script that runs forever, starts on boot, and it's job is to launch, and once every minute check the status of your workers using process script, and locate if they died and relaunch. The logic is yours, you are in control how you want to manage this, but keep it simple. This broker script can run forever from CLI and should not have a memory leaking.

Upvotes: 1

Related Questions