NetMsmqBinding: messages are being processed multiple times

Question

My team has a windows service which host a wcf service using http binding. When the wcf service receives a message, it immediately sends it to another wcf service, using NetMsmqBinding, also hosted in a windows service. The msmq service process the message and save it do our DB.

In the development and staging environments, everything worked fine. When we deployed to production though, we saw that messages sent to our service are being multiplied by a factor of n+1 (for n messages).

For example: 1 message received, 2 saved to DB. 4 messages received, 20 saved to DB etc.

We suspect that it has something to do with the msmq service settings of Transactions,

Concurrency, and InstanceContextMode, but we can't figure it out.

The settings of the service are:

• InstanceContextMode: Default.
• ConcurrenyMode: Default
• Transaction: TransactionScoreRequired = True and TransactionAutoComplete = True.
• RetryCount = 1, RetryCycles = 1

We use .Net 4, MSMQ 3.0 and Windows server 2003.

Any ideas to what might cause this issue?

Edit:

After a lot of research, we have discovered the following things:

• When the wcf runtime tries to commit the transaction for a single message, it throws an exception of "Transaction was aborted asynchronously". This exception is thrown each time a message is duplicated.
• In the wcf trace, we don't see any exception other then the one stated above.
• In the System.Transaction trace, we see a transaction is being opened (for the 1st message), and then after the wcf runtime creates a clone of that transaction (with the property RollbackIfNotCompleted=true), a few seconds pass without no message in the trace and then the same thing happens again.

We really are clueless...

Thanks

Answer 1

Well, we finally got it: we use Entlib for logging, and we added a DB listener. Entlib DB listener uses the same transaction of the service. When it tries to pass the transaction to the DB server, an error is thrown because MSDTC wasn't configured correctly on the DB server, which causes the transaction to rollback. When the WCF runtime tries to commit the transaction, it fails because it was already aborted, and then it tries to process the same message again (because of the built-in retry mechanism).

When we fixed the MSDTC settings, the duplication stopped.
Now, all we need to figure out is: why rollback to the service transaction causes messages to be multiplied so many times?

Answer 2

MSMQ won't remove a message unless either your application removes it or the TimeToBeReceived timer expires.

If I understood your environment correctly, your problem might be caused by a too long processing time of a received message. If the receiving and re-sending of a message is done within the same transaction it might be possible that the first WCF service tries to read the same message multiple times before it is finally removed from the queue.

This leads me to the following ideas you can try:

• You might try, after receiving your original message, to create a new message with just the same contents and send the new one to the next WCF service. This should reduce the life-time of the received messages.

• If you need to retain the same order of the messages you can try Miker169's suggestion to change your configuration into InstanceContextMode.Single , ConcurrencyMode = ConcurrencyMode.Single (https://stackoverflow.com/a/2610093/219344)