Reputation: 644
For background information, we are a retailer company with more than 200 stores. We use rabbitmq federation with 200+ RabbitMQ servers (one server in each store). Each RabbitMQ server has 4 exchanges. So there are around 800 exchanges in total.
There is one central server where messages are published to. This is used as the upstream server for the other 200 servers. When there is a product change, a message is sent to the central server which will be forwarded to other rabbitmq servers and is processed locally in each store. Our application runs on .NET framework on windows servers (using Masstransit to handle RabbitMQ).
Sometimes, we get timeout errors when publishing events to rabbitMQ. This is an example:
Properties
message_id: f68a0000-15f3-0670-085b-08d66b92a0b1
delivery_mode: 2
headers:
x-correlation-id: 18799970-fea8-431e-8608-d02496194a0a
Content-Type: application/vnd.masstransit+json
publishId: 1
MT-Reason: fault
MT-Fault-Message: Publishing message of type MyCompany.Event.Message timed out after 5000 milliseconds.
MT-Fault-Timestamp: 2018-12-27T00:32:33.6076503Z
MT-Fault-StackTrace: at MyCompany.ProductCommand.Listener.UpsertProductCommandListener.<PublishMessage>d__10`1.MoveNext()
at MyCompany.ProductCommand.Listener.UpsertProductCommandListener.<PublishUpdatedEvents>d__8.MoveNext()
at MyCompany.ProductCommand.Listener.UpsertProductCommandListener.<Consume>d__7.MoveNext()
at MassTransit.AutofacIntegration.AutofacConsumerFactory`1.<Send>d__3`1.MoveNext()
at MassTransit.Pipeline.Filters.ConsumerMessageFilter`2.<GreenPipes-IFilter<MassTransit-ConsumeContext<TMessage>>-Send>d__4.MoveNext()
at MassTransit.Pipeline.Filters.ConsumerMessageFilter`2.<GreenPipes-IFilter<MassTransit-ConsumeContext<TMessage>>-Send>d__4.MoveNext()
at GreenPipes.Filters.TeeFilter`1.<Send>d__5.MoveNext()
at GreenPipes.Filters.OutputPipeFilter`2.<GreenPipes-IFilter<TInput>-Send>d__6.MoveNext()
at GreenPipes.Filters.OutputPipeFilter`2.<GreenPipes-IFilter<TInput>-Send>d__6.MoveNext()
at MassTransit.Pipeline.Filters.DeserializeFilter.<Send>d__4.MoveNext()
at GreenPipes.Filters.RescueFilter`2.<GreenPipes-IFilter<TContext>-Send>d__5.MoveNext()
MT-Host-MachineName: WIN-123456
MT-Host-ProcessName: MyCompany.ProductCommandListener.WindowsService
MT-Host-ProcessId: 848
MT-Host-Assembly: MyCompany.ProductCommandListener.WindowsService
MT-Host-AssemblyVersion: 1.0.0.0
MT-Host-MassTransitVersion: 3.5.4.992
MT-Host-FrameworkVersion: 4.0.30319.42000
MT-Host-OperatingSystemVersion: Microsoft Windows NT 6.2.9200.0
content_type: application/vnd.masstransit+json
Each message is around 5000 bytes. We would like to understand how the message got timeout. We found this in rabbitMQ website regarding ack behaviour:
For routable messages, the basic.ack is sent when a message has been accepted by all the queues. For persistent messages routed to durable queues, this means persisting to disk. For mirrored queues, this means that all mirrors have accepted the message.
(https://www.rabbitmq.com/confirms.html#when-publishes-are-confirmed)
However, we couldn't find any information regarding publishing acknowledgement behavior in federation. So our questions are
Any help would be appreciated.
Upvotes: 1
Views: 722