Beanstalkd (via pheanstalk) allowing duplicate, simultaneous reserves?

Question

Trying to wrap Pheanstalk in my PHP job base class. I'm testing the reserve and reserve with delay functionality and I've found that I can reserve a job from a second instances of my base class without the first instance releasing the job or the TTR timing out. This is unexpected since I was thinking this is exactly the thing job queues are supposed to prevent. Here are the beanstalkd commands for the first put and the first reserve along with time stamps. I also do a stats-job request at the end:

01:40:15: Sending command: use QueuedCoreEvent
01:40:15: Got response: USING QueuedCoreEvent

01:40:15: Sending command: put 1024 0 300 233
a:4:{s:9:"eventName";s:21:"ReQueueJob_eawu7xr9bi";s:6:"params";a:2:{s:12:"InstanceName";s:21:"ReQueueJob_eawu7xr9bi";s:17:"aValueToIncrement";i:123456;}s:9:"behaviors";a:1:{i:0;s:22:"BehMCoreEventTestDummy";}s:12:"failureCount";i:0;}
01:40:15: Got response: INSERTED 10

01:40:15: Sending command: watch QueuedCoreEvent
01:40:15: Got response: WATCHING 2

01:40:15: Sending command: ignore default
01:40:15: Got response: WATCHING 1

01:40:15: Sending command: reserve-with-timeout 0
01:40:15: Got response: RESERVED 10 233
01:40:15: Data: a:4:{s:9:"eventName";s:21:"ReQueueJob_eawu7xr9bi";s:6:"params";a:2:{s:12:"InstanceName";s:21:"ReQueueJob_eawu7xr9bi";s:17:"aValueToIncrement";i:123456;}s:9:"behaviors";a:1:{i:0;s:22:"BehMCoreEventTestDummy";}s:12:"failureCount";i:0;}

01:40:15: Sending command: stats-job 10
01:40:15: Got response: OK 162
01:40:15: Data: ---
id: 10
tube: QueuedCoreEvent
state: reserved
pri: 1024
age: 0
delay: 0
ttr: 300
time-left: 299
file: 0
reserves: 1
timeouts: 0
releases: 0
buries: 0
kicks: 0

So far, so good. Now I do another reserve from a second instance of my base class followed by another stats-job request. Notice the time stamps are within the same second, nowhere near the 300 second TTR I've set. Also notice in this second stats-job printout that there are 2 reserves of this job with 0 timeouts and 0 releases.

01:40:15: Sending command: watch QueuedCoreEvent
01:40:15: Got response: WATCHING 2

01:40:15: Sending command: ignore default
01:40:15: Got response: WATCHING 1

01:40:15: Sending command: reserve-with-timeout 0
01:40:15: Got response: RESERVED 10 233
01:40:15: Data: a:4:{s:9:"eventName";s:21:"ReQueueJob_eawu7xr9bi";s:6:"params";a:2:{s:12:"InstanceName";s:21:"ReQueueJob_eawu7xr9bi";s:17:"aValueToIncrement";i:123456;}s:9:"behaviors";a:1:{i:0;s:22:"BehMCoreEventTestDummy";}s:12:"failureCount";i:0;}

01:40:15: Sending command: stats-job 10
01:40:15: Got response: OK 162
01:40:15: Data: ---
id: 10
tube: QueuedCoreEvent
state: reserved
pri: 1024
age: 0
delay: 0
ttr: 300
time-left: 299
file: 0
reserves: 2
timeouts: 0
releases: 0
buries: 0
kicks: 0

Anyone have any ideas on what I might be doing wrong? Is there something I have to do to tell the queue I want jobs to only be accessed by one worker at a time? I'm doing an "unset" on the pheanstalk instance as soon as I get the job off the queue which I believe terminates the session with beanstalkd. Could this cause beanstalkd to decide the worker has died and automatically release the job without a timeout? I'm uncertain of how much beanstalkd relies on session state to determine worker state. I was assuming that I could open and close sessions with impunity and that job id was the only thing that beanstalkd cared about to tie job operations together, but that may have been foolish on my part... This is my first foray into job queues.

Thanks!

Answer 1

I got the same issue. The problem was in multiple connections opened to beanstalkd.

use Pheanstalk\Pheanstalk;

$pheanstalk = connect();
$pheanstalk->put(serialize([1]), 1, 0, 1800);

/** @var Job $job */
$job = $pheanstalk->reserve(10);
print_r($pheanstalk->statsJob($job->getId()));
// state reserved but
// only those connection that reserved a job can resolve/update it

$pheanstalk2 = connect();
print_r($pheanstalk->statsJob($job->getId()));
$pheanstalk2->delete($job);
// new connection opened in same process still cannot update the job 

// PHP Fatal error:  Uncaught Pheanstalk\Exception\ServerException: Cannot delete job 89: NOT_FOUND in /var/www/vendor/pda/pheanstalk/src/Command/DeleteCommand.php:45


function connect() {
    $pheanstalk = new Pheanstalk(
        'localhost',
        11300,
        5
    );
    return $pheanstalk;
}

Answer 2

My guess is your first client instance closed the TCP socket to the beanstalkd server before the second one reserved the job.

Closing the TCP connection implicitly releases the job back onto the queue. These implicit releases (close connection, quit command etc) do not seem to increment the releases counter.

Here's an example:

# Create a job, reserve it, close the connection:
pda@paulbookpro ~ > telnet 0 11300
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
put 0 0 600 5
hello
INSERTED 1
reserve
RESERVED 1 5
hello
^]
telnet> close
Connection closed.

# Reserve the job, stats-job shows two reserves, zero releases.
# Use 'quit' command to close connection.
pda@paulbookpro ~ > telnet 0 11300
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
reserve
RESERVED 1 5
hello
stats-job 1
OK 151
---
id: 1
tube: default
state: reserved
pri: 0
age: 33
delay: 0
ttr: 600
time-left: 593
file: 0
reserves: 2
timeouts: 0
releases: 0
buries: 0
kicks: 0

quit
Connection closed by foreign host.

# Reserve the job, stats-job still shows zero releases.
# Explicitly release the job, stats-job shows one release.
pda@paulbookpro ~ > telnet 0 11300
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
reserve
RESERVED 1 5
hello
stats-job 1
OK 151
---
id: 1
tube: default
state: reserved
pri: 0
age: 46
delay: 0
ttr: 600
time-left: 597
file: 0
reserves: 3
timeouts: 0
releases: 0
buries: 0
kicks: 0

release 1 0 0
RELEASED
stats-job 1
OK 146
---
id: 1
tube: default
state: ready
pri: 0
age: 68
delay: 0
ttr: 600
time-left: 0
file: 0
reserves: 3
timeouts: 0
releases: 1
buries: 0
kicks: 0

quit
Connection closed by foreign host.