CODEWITHSUNDEEP
ignitegridgain
Ishan Gandhi
Ishan Gandhi

Reputation: 47

Optimizing Data Streamer Pool size to improve ignite performance

I am using Ignite 2.6 with data streamer nodes consuming data from kafka and putting in ignite cache. A high server load average is being observed with reduced throughput.

I have tried giving index inline size for the indexes defined in the cache which is giving a good performance but also increases the server memory utilization and high load average. Please advise how does increasing the datastreamer threadpool size will impact in this case.

Upvotes: 0

Views: 216

Answers (1)

Denis Mekhanikov
Denis Mekhanikov

Reputation: 3591

When optimizing data streamer performance, you need to understand, where the bottleneck is. It may be either on data nodes or streaming nodes. Servers may be inserting data slowly or streamers may be just giving too little load. CPU, memory, network, disk resources can be drained on either side. Therefore, before trying to optimize configuration parameters, try to look for a reason, that's causing the slowness. Using JFR, JProfiler, VisualVM or something similar may help you with it.

If the bottleneck is on the streaming side, then in most cases calling IgniteDataStreamer#addData on a single data streamer from multiple threads increases servers' utilization.

If you see the issue on the server's side, then you may need to increase the number of servers or try to optimize the insertion time.

Upvotes: 1

Related Questions