CODEWITHSUNDEEP
javacassandrawebspherecassandra-3.0
ANIRBAN GHOSH
ANIRBAN GHOSH

Reputation: 65

Cassandra performance is different with respect to two application host

I have my java application that writes bulk records into cassandra cluster.My application is running on two WAS nodes and both connected to same cluster.The bulk insert (asynchronous) process runs simultaneously on both the was nodes. The 1st WAS node inserts the first 50% of the entire set of records to be inserted and the 2nd WAS node inserts the rest 50%.

We observed one of these two server nodes is taking almost double time to complete its insertion process. Both the WAS instance has sane configuration and same cluster connected.

Please suggest what would be possible reason.

Upvotes: 0

Views: 55

Answers (1)

Gary DeVal
Gary DeVal

Reputation: 301

One possibility is that the two WAS nodes are not identical, although you intended them to be. Basic platform performance information is the starting point for diagnosing any performance problem.

I would start diagnosing this problem by comparing performance monitoring information for the two WAS nodes, using tooling like NMON if you are on Linux.

http://nmon.sourceforge.net/pmwiki.php

A command like this will cause nmon data to be written to a file every 10 seconds for 1800 samples 

nmon -f -F <filename.nmon> -s 10 -c 1800 -t

and then with a tool like NMON visualizer you can compare the data for the two nodes in graphical form

https://nmonvisualizer.github.io/nmonvisualizer/

Similar tooling is available for other platforms, e.g. perfmon for Windows.

In this case I would be looking initially for a difference in CPU used between the two WAS nodes. If CPU is high in the node that takes longer to insert the records, maybe that node is configured with fewer cores (VM), or maybe the java heap in that node is smaller so it is spending lots of time doing Garbage Collection, or maybe that node was configured with SSL and the other node was not, etc.etc.

If CPU is low in the node that takes longer to insert the records, then there must be some external bottleneck that is limiting the work done by the node - maybe the network port on the node is misconfigured so that traffic between that node and the Cassandra cluster is limited, or maybe the configuration of the Cassandra interface for that node is incorrect, or maybe the node has a slow or failing hard drive so reading the bulk data for insert is slow, etc.etc.

Performance problem diagnosis requires gathering performance data and then following the clues.

Upvotes: 1

Related Questions