CODEWITHSUNDEEP
azure-service-fabric
Michiel Overeem
Michiel Overeem

Reputation: 3982

Can we prevent deadlocks and timeouts on ReliableQueue's in Service Fabric?

We have a stateful service in Service Fabric with both a RunAsync method and a couple of service calls.

One service calls allows to enqueue something in a ReliableQueue

using(ITransaction tx = StateManager.CreateTransaction())
{
  await queue.EnqueueAsync(tx, message);
  queueLength = await queue.GetCountAsync(tx);
  await tx.CommitAsync();
}

The RunAsync on the other hand tries to dequeue things:

using(ITransaction tx = StateManager.CreateTransaction())
{
  await queue.TryDequeueAsync(tx);
  queueLength = await queue.GetCountAsync(tx);
  await tx.CommitAsync();
}

The GetCountAsync seems to cause deadlocks, because the two transactions block each other. Would it help if we would switch the order: so first counting and then the dequeue/enqueue?

Upvotes: 2

Views: 372

Answers (2)

masnider
masnider

Reputation: 2599

This is likely due to the fact that the ReliableQueue today is strict FIFO and allows only one reader or writer at a time. You're probably not seeing deadlocks, you're seeing timeouts (please correct me if that is not the case). There's no real way to prevent the timeouts other than to:

  • Ensure that the transactions are not long lived - any longer than you need and you're blocking other work on the queue.
  • Increase the default transaction timeout (the default is 4 seconds, you can pass in a different value)

Reordering things shouldn't cause any change.

Upvotes: 1

pdylanross
pdylanross

Reputation: 275

Having two transactions in two different places shouldn't cause deadlocks, as they act like mutexes. What will cause them though is creating transactions within transactions.

Perhaps that is what's happening? I've developed the habit lately of naming functions that create transactions Transactional, ie DoSomethingTransactionalAsync, and if it's a private helper I'll usually create two versions with one taking a tx and one creating a tx.

For example:

AddToProcessingQueueAsync(ITransaction tx, int num) and AddToProcessingQueueTransactionalAsync(int num).

Upvotes: 1

Related Questions