CODEWITHSUNDEEP
cassandra
Natasha
Natasha

Reputation: 9

TWCS is not generating the SStables according to the window_size and window_unit

I need to generate the SSTables after a certain time like 10 minutes, but using TWCS and setting up the "compaction_window_size" and "compaction_window_unit", I am not able to understand when the SSTables would be getting generated.

I have tried all many combinations but I am not able to figure out the when the SSTables will be created

CREATE TABLE twcs.twcs2 (
    id int PRIMARY KEY,
    age int,
    name text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 'compaction_window_size': '1', 'compaction_window_unit': 'MINUTES', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 3600
    AND gc_grace_seconds = 60
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

Here I have set the 'compaction_window_unit'='MINUTES' and 'compaction_window_size': '1' , So according to this the SSTables should be generated after every 1 minute if some operation is performed on the table(insertion/deletion/updation of data), but this is not happening.

Upvotes: 0

Views: 367

Answers (2)

Jim Wartnick
Jim Wartnick

Reputation: 2196

TWCS is a compaction strategy. Compaction strategies have nothing to do with sstables being generated. It's a reconcile and cleanup "algorithm" once they are created. The way that TWCS works is that sstables will be consolidated into windows. The key word here is "consolidated". There is no guarantee that sstables will be "generated" in that timeframe, but whatever IS generated, after the window expires, will be consolidated together. So if you have, say, hourly windows/buckets, during that hour sstables may or may not get generated. If multiple sstables are created during the window they are compacted/consolidated/reconciled using STCS (similar sized sstables consolidated together). After the hour passes, any sstables that remain for that window will be compacted together into a single sstable. Over time you will see one sstable per window (or none if nothing was generated during that window). After your TTL and gc_grace passes, entire windows simply get removed (instead of the large effort of merging with others and then removing expired records).

TWCS works very well if there is no overlap for rows within windows. If there is overlap, then the oldest windowed sstable will be unable to be removed until the newest sstable with overlapped records expire. In other words, TWCS works well for INSERTS that do not cross windows (remember updates and deletes are also considered inserts). You need to be sure to use TTL for cleanup (i.e. don't run delete statements as that will cause overlap). Also, from what I have discovered from using this, ensure to turn off repair for the tables that have TWCS as that can cause big problems (unseen overlap).

So in short, TWCS does not cause sstables to get generated (there are rules for when sstables are created that have nothing to do with compaction strategies), it simply another method of keep things "clean".

Hope that helps.

Upvotes: 1

Carlos Monroy Nieblas
Carlos Monroy Nieblas

Reputation: 2283

There are several resources that may help you:

Upvotes: 0

Related Questions