CODEWITHSUNDEEP
spring-integrationspring-xd
constantlearner
constantlearner

Reputation: 5247

Spring xd performance with message size

I am finding below observations in spring xd

I have a stream like below 

s3-source| processor-parser|http-client1|Request-construct-processor>queue

queue > processor|http-client2|processor|http-client3>queue

When my http-client output json data of 2 kb i am able to process more message like 20k msg per min. If data size is 26kb its so slow that it process 2000 msg per min.Request-construct-processor create json using response from http and output string json.We have increased channel cache size to 1000.Currently my rabbit cpu utilization goes to 100% and it moves to flow state? Also as msg size increases I see slower msg rate and rabbit cpu increase?is there any way to over come this.

My rabbit cpu is high and it moves to flow state. Msg rate has slowed down with increase in message size. I am processing over million messages 3 container one admin and 3 rabbit all with 8 cpu and 32 gb ram machines. I am running all module with concurrency with 10 I am also passing http1 response also in message header from Request-construct-processor Also i see lot of msg in transport queues in between processors and they are in idle stage and xd stops processing.

Update 2

I was passing 26 kb in header and I removed it now i see good performance.

Upvotes: 0

Views: 66

Answers (1)

Ilayaperumal Gopinathan
Ilayaperumal Gopinathan

Reputation: 4179

I wonder if you can make use of batchSize and batchTimeout properties for Rabbit message bus that could improve the performance.

For more detail you can refer here

Upvotes: 1

Related Questions