CODEWITHSUNDEEP
cassandracassandra-3.0
Igor
Igor

Reputation: 3

Issue num_tokens on cassandra after perform a node restore - Cassandra 3.11.10 and 3.11.11

After upgrading from 3.0.13 to 3.11.10 I'm having issue when I retsore the data from the node. After the restore is done it changes the number of tokens from 256 to 512.In cassandra.yaml it still num_token: 256, so I have no idea what's going on here.

These are the steps I'm performing

rm -rf /var/lib/cassandra/data/*
rm -rf /var/lib/cassandra/commitlog/*
rm -rf /var/lib/cassandra/saved_caches/*
rm -rf /var/lib/cassandra/hints/*
rm -rf /var/log/cassandra/*
tar -xvf $BACKUP_LOC/$BACKUP_NAME.tar -C /
find ${DATA_DIR} -mindepth 2 -path "*/snapshots/${BACKUP_NAME}/*" -type f \-exec bash -c 'dir={} && cd ${dir%/*} && mv {} ../..' \;

nodetool -u $NODETOOL_USR -pw $NODETOOL_PASSWD -h $(hostname) refresh $keyspace $table --ssl" >>$BACKUP_LOC/nodetool_refresh_commands.sh

When I restart the node after perfoming the steps above it changes the tokens from 256 to 512.

ERROR:

ERROR [main] 2021-10-07 15:16:24,060 CassandraDaemon.java:803 - Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Cannot change the number of tokens from 512 to 256 at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1102) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:760) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:694) ~[apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:395) [apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:633) [apache-cassandra-3.11.11.jar:3.11.11] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:786) [apache-cassandra-3.11.11.jar:3.11.11] INFO [StorageServiceShutdownHook] 2021-10-07 15:16:24,064 HintsService.java:209 - Paused hints dispatch INFO [StorageServiceShutdownHook] 2021-10-07 15:16:24,064 Gossiper.java:1683 - Announcing shutdown

Any ideas on it? Thanks,

Upvotes: 0

Views: 478

Answers (1)

Erick Ramirez
Erick Ramirez

Reputation: 16353

The problem here is that you've deleted all the contents of the data/ directory including the system keyspaces/tables:

rm -rf /var/lib/cassandra/data/*

You shouldn't restore system tables from backups -- you should only ever restore application keyspaces/tables.

The most likely cause of this exception:

ERROR [main] 2021-10-07 15:16:24,060 CassandraDaemon.java:803 - \
  Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: \
  Cannot change the number of tokens from 512 to 256

is that the system.local table got corrupted.

If you try to dump the contents of one of the SSTables for system.local, you'll see that it will contain 512 tokens. For example:

$ sstabledump me-1234-big-Data.db
...
          { "name" : "tokens", "path" : [ "-1090765148586270517" ], "value" : "", "tstamp" : "2021-10-12T01:23:45.678Z" },
          { "name" : "tokens", "path" : [ "-1118040538640756774" ], "value" : "", "tstamp" : "2021-10-12T01:23:45.678Z" },
          { "name" : "tokens", "path" : [ "-1256255083036528868" ], "value" : "", "tstamp" : "2021-10-12T01:23:45.678Z" },
...

$ sstabledump me-1234-big-Data.db | grep -c tokens
512

There is no workaround available once the system.local table gets corrupted. The only solution is to completely wipe all the C* directories and re-bootstrap the node. Cheers!

Upvotes: 2

Related Questions