Advise on stopping compaction to reduce slowness

Question

I am seeing high CPU and memory usage of cassandra on the seed node. Is it advisable to stop compaction(nodetool stop) and enable in offpeak hours. Should I do manual compaction or enable autocompaction. I see lot of Native-Transport-Requests. I have three seed nodes. This is the first seed node.

Pool Name                         Active   Pending      Completed   Blocked  All time blocked
ReadStage                              0         0          54255         0                 0
MiscStage                              0         0              0         0                 0
CompactionExecutor                     2      2566         352765         0                 0
MutationStage                          0         0     2659921760         0                 0
MemtableReclaimMemory                  0         0         180958         0                 0
PendingRangeCalculator                 0         0             21         0                 0
GossipStage                            0         0         338375         0                 0
SecondaryIndexManagement               0         0              0         0                 0
HintsDispatcher                        0         0             63         0                 0
RequestResponseStage                   0         1     1684328696         0                 0
Native-Transport-Requests              4         0     1538523706         0          47006391
ReadRepairStage                        0         0           2197         0                 0
CounterMutationStage                   0         0              0         0                 0
MigrationStage                         0         0              0         0                 0
MemtablePostFlush                      1         1         216220         0                 0
PerDiskMemtableFlushWriter_0           1         1         180958         0                 0
ValidationExecutor                     0         0          33250         0                 0
Sampler                                0         0              0         0                 0
MemtableFlushWriter                    1         1         180958         0                 0
InternalResponseStage                  0         0         141677         0                 0
ViewMutationStage                      0         0              0         0                 0
AntiEntropyStage                       0         0         166254         0                 0
CacheCleanupExecutor                   0         0              0         0                 0
Repair#9                               0         0           5719         0                 0

I do see high compactions. Is it advisable to disable compactions using nodetool stop

$ nodetool info
ID                     : ebeda774-cea8-40bb-9322-69c6fcded5a9
Gossip active          : true
Thrift active          : true
Native Transport active: true
Load                   : 535.37 GiB
Generation No          : 1636316595
Uptime (seconds)       : 73152
Heap Memory (MB)       : 19542.18 / 32168.00
Off Heap Memory (MB)   : 1337.98
Data Center            : us-west2
Rack                   : a
Exceptions             : 15
Key Cache              : entries 152283, size 23.07 MiB, capacity 100 MiB, 23835 hits, 280738 requests, 0.085 recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache            : entries 6782, size 423.88 MiB, capacity 480 MiB, 23947952 misses, 24381819 requests, 0.018 recent hit rate, 250.977 microseconds miss latency
Percent Repaired       : 0.49796724500672584%
Token                  : (invoke with -T/--tokens to see all 256 tokens)

$ free -h
              total        used        free      shared  buff/cache   available
Mem:            62G         53G        658M        1.0M        8.5G        8.5G
Swap:            0B          0B          0B

~$ nodetool compactionstats
pending tasks: 197
....
id                                   compaction type keyspace         table                          completed total    unit  progress
5e555610-40b2-11ec-9b5a-27bc920e6e55 Compaction      mykeyspace       table1 27299674  89930474 bytes 30.36%
5e55f251-40b2-11ec-9b5a-27bc920e6e55 Compaction      mykeyspace       table2 13922048  74426264 bytes 18.71%
Active compaction remaining time :   0h00m02s

Answer 1

I would definitely not run compaction manually. Much of the compaction thresholds are file-size based, which means that forcing it creates files sized outside of the normal progression. The result, is that the chances of compaction running on that table again are extremely slim. Basically, once you start down that path, you'll be running manual compactions forever.

I would also say that compaction is a good thing. You want it to happen, as compacted files are necessary to keep reads performing well. Of course, that's not much of a consolation when the compaction process is affecting operational activity.

tl;dr;

One I have done in the past, is to lower compaction throughput during the day. Not sure what throughput you're running with currently, but you can find this out by running nodetool getcompactionthroughput:

% bin/nodetool getcompactionthroughput
Current compaction throughput: 64 MB/s

So at the times when customer/operational traffic is high, you can reduce that significantly:

% bin/nodetool setcompactionthroughput 1
% bin/nodetool getcompactionthroughput
Current compaction throughput: 1 MB/s

1 MB / second is the lowest that compaction throughput can be set. If you set it to zero, it's "un-throttled," which means it'll consume all the resources that it can get at. Setting it to 1 brings its resource use (and speed) down to a trickle.

Once the busy daily traffic subsides, that setting can be turned back up:

% bin/nodetool setcompactionthroughput 256
Current compaction throughput: 256 MB/s

This can be accomplished with a scheduled job for each command.