CODEWITHSUNDEEP
cassandrajvm
nmakb
nmakb

Reputation: 1245

how are cassandra sessions allocated

I see cassandra java process on my linux instance, it uses ~38gb of memory and shows ~700 threads under it.

  1. When any connections are made to the database via python or java? Do they become a thread under the main java process or a separate OS process?
  2. What happens when a cluster connection is spawning multiple threads, would they also become threads under the main process? If so, how to distinguish between connection threads and connection spawning threads?
  3. The memory allocated for session threads, does it get allocated under non-heap memory?

Update - @chris - here is the output of tpstats

[username@hostname ~]$ nodetool tpstats
Pool Name                         Active   Pending      Completed   Blocked  All time blocked
ReadStage                              0         0      110336013         0                 0
ContinuousPagingStage                  0         0             31         0                 0
MiscStage                              0         0              0         0                 0
CompactionExecutor                     0         0        4244757         0                 0
MutationStage                          0         0       25309020         0                 0
GossipStage                            0         0        2484700         0                 0
RequestResponseStage                   0         0       46705216         0                 0
ReadRepairStage                        0         0        2193356         0                 0
CounterMutationStage                   0         0        3563130         0                 0
MemtablePostFlush                      0         0         117717         0                 0
ValidationExecutor                     1         1         111176         0                 0
MemtableFlushWriter                    0         0          23843         0                 0
ViewMutationStage                      0         0              0         0                 0
CacheCleanupExecutor                   0         0              0         0                 0
Repair#1953                            1         3              1         0                 0
MemtableReclaimMemory                  0         0          28251         0                 0
PendingRangeCalculator                 0         0              6         0                 0
AntiCompactionExecutor                 0         0              0         0                 0
SecondaryIndexManagement               0         0              0         0                 0
HintsDispatcher                        0         0             29         0                 0
Native-Transport-Requests              0         0      110953286         0                 0
MigrationStage                         0         0             19         0                 0
PerDiskMemtableFlushWriter_0           0         0          27853         0                 0
Sampler                                0         0              0         0                 0
InternalResponseStage                  0         0          21264         0                 0
AntiEntropyStage                       0         0         350913         0                 0

Message type           Dropped                  Latency waiting in queue (micros)                                    
                                             50%               95%               99%               Max
READ                         0              0.00              0.00              0.00          10090.81
RANGE_SLICE                  0              0.00              0.00          10090.81          10090.81
_TRACE                       0               N/A               N/A               N/A               N/A
HINT                         0              0.00              0.00              0.00              0.00
MUTATION                     0              0.00              0.00              0.00          10090.81
COUNTER_MUTATION             0              0.00              0.00              0.00          10090.81
BATCH_STORE                  0              0.00              0.00              0.00              0.00
BATCH_REMOVE                 0              0.00              0.00              0.00              0.00
REQUEST_RESPONSE             0              0.00              0.00              0.00          12108.97
PAGED_RANGE                  0               N/A               N/A               N/A               N/A
READ_REPAIR                  0              0.00              0.00              0.00              0.00```

Upvotes: 1

Views: 138

Answers (1)

Chris Lohfink
Chris Lohfink

Reputation: 16410

The connections go to a netty service which should have threads equal to the number of cores, even if you have 10000 connected clients. However Cassandra was initially designed with a Staged Event Driven Architecture (SEDA) which kinda sits between an async and fully threaded model. It creates pools of threads to handle different types of tasks. This does mean however depending on the configuration in your yaml there can be a lot of threads. For example by default theres up to 128 threads for the native transport pool, 32 concurrent readers, 32 concurrent writers, 32 counter mutations etc but if your cluster was tuned for ssds these might be higher. That in mind theres a number of these pools that use a shared pool (show up as SharedWorkers) with the SEPExecutor (Single executor pool). So with spikes there might be many created but the threads may not be utilized often.

nodetool tpstats will give you details on the different pools and how many are active which can help identify which and if the threads are being used. If not there then you can also use jstack (use as same user as the cassandra process) to dump the traces. If its too much to look through theres tools like https://fastthread.io/ to make viewing it easier.

For what it is worth 32gb of memory and 700 threads doesn't sound like an issue.

Upvotes: 1

Related Questions