Ruby with_advisory_lock test with multiple threads fails intermittently

Question

I'm using the with_advisory_lock gem to try and ensure that a record is created only once. Here's the github url to the gem.

I have the following code, which sits in an operation class that I wrote to handle creating user subscriptions:

def create_subscription_for user
  subscription = UserSubscription.with_advisory_lock("lock_%d" % user.id) do
    UserSubscription.where({ user_id: user.id }).first_or_create
  end

  # do more stuff on that subscription
end

and the accompanying test:

threads = []
user = FactoryBot.create(:user)

rand(5..10).times do
  threads << Thread.new do
    subject.create_subscription_for(user)
  end
end

threads.each(&:join)

expect(UserSubscription.count).to eq(1)

What I expect to happen:

• The first thread to get to the block acquires the lock and creates a record.
• Any other thread that gets to the block while it's being held by another thread waits indefinitely until the lock is released (as per docs)
• As soon as the lock is released by the first thread that created the record, another thread acquires the lock and now finds the record because it was already created by the first thread.

What actually happens:

• The first thread to get to the block acquires the lock and creates a record.
• Any other thread that gets to the block while it's being held by another thread goes and executes the code in the block anyway and as a result, when running the test, it sometimes fails with a ActiveRecord::RecordNotUnique error (I have a unique index on the table that allows for a single user_subscription with the same user_id)

What is more weird is that if I add a sleep for a few hundred milliseconds in my method just before the find_or_create method, the test never fails:

def create_subscription_for user
  subscription = UserSubscription.with_advisory_lock("lock_%d" % user.id) do
    sleep 0.2
    UserSubscription.where({ user_id: user.id }).first_or_create
  end

  # do more stuff on that subscription
end

My questions are: "Why is adding the sleep 0.2 making the tests always pass?" and "Where do I look to debug this?"

Thanks!

UPDATE: Tweaking the tests a little bit causes them to always fail:

threads = []
user = FactoryBot.create(:user)

rand(5..10).times do
  threads << Thread.new do
    sleep
    subject.create_subscription_for(user)
  end
end

until threads.all? { |t| t.status == 'sleep' }
  sleep 0.1
end

threads.each(&:wakeup)
threads.each(&:join)

expect(UserSubscription.count).to eq(1)

I have also wrapped first_or_create in a transaction, which makes the test pass and everything to work as expected:

def create_subscription_for user
  subscription = UserSubscription.with_advisory_lock("lock_%d" % user.id) do
    UserSubscription.transaction do
      UserSubscription.where({ user_id: user.id }).first_or_create
    end
  end

  # do more stuff on that subscription
end

So why is wrapping first_or_create in a transaction necessary to make things work?

Answer 1

Are you turning off transactional tests for this test case? I'm working on something similar and that proved to be important to actually simulating the concurrency.

See uses_transaction https://api.rubyonrails.org/classes/ActiveRecord/TestFixtures/ClassMethods.html

If transactions are not turned off, Rails will wrap the entire test in a transaction and this will cause all the threads to share one DB connection. Furthermore, in Postgres a session-level advisory lock can always be re-acquired within the same session. From the docs:

If a session already holds a given advisory lock, additional requests by it will always succeed, even if other sessions are awaiting the lock; this statement is true regardless of whether the existing lock hold and new request are at session level or transaction level.

Based on that I'm suspecting that your lock is always able to be acquired and therefore the .first_or_create call is always executed which results in the intermittent RecordNotUnique exceptions.