CODEWITHSUNDEEP
javacassandracassandra-4.0
Jia Bochao
Jia Bochao

Reputation: 23

cassandra-4.0-beta4 Exception encountered during startup Requested permits (0) must be positive

Error message:

ERROR [main] 2021-03-25 14:16:15,561 CassandraDaemon.java:822 - Exception encountered during startup
org.apache.cassandra.exceptions.TruncateException: Error during truncate: java.lang.IllegalArgumentException: Requested permits (0) must be positive
    at org.apache.cassandra.cql3.statements.TruncateStatement.executeLocally(TruncateStatement.java:96)
    at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:323)
    at org.apache.cassandra.db.SystemKeyspace.clearAllEstimates(SystemKeyspace.java:1337)
    at org.apache.cassandra.service.StorageService.cleanupSizeEstimates(StorageService.java:3908)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:676)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:800)
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Requested permits (0) must be positive
    at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2329)
    at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2276)
    at org.apache.cassandra.db.ColumnFamilyStore.truncateBlocking(ColumnFamilyStore.java:2256)
    at org.apache.cassandra.cql3.statements.TruncateStatement.executeLocally(TruncateStatement.java:92)
    ... 6 common frames omitted
Caused by: java.lang.IllegalArgumentException: Requested permits (0) must be positive
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:189)
    at com.google.common.util.concurrent.RateLimiter.checkPermits(RateLimiter.java:430)
    at com.google.common.util.concurrent.RateLimiter.reserve(RateLimiter.java:285)
    at com.google.common.util.concurrent.RateLimiter.acquire(RateLimiter.java:273)
    at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1801)
    at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1981)
    at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1957)
    at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1945)
    at org.apache.cassandra.db.ColumnFamilyStore$4.run(ColumnFamilyStore.java:2242)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2325)
    ... 9 common frames omitted

Upvotes: 1

Views: 1353

Answers (4)

Shane Gannon
Shane Gannon

Reputation: 7708

Noticed the same problem with a docker container. When I checked the logs

docker logs 59d4e2890a34

it highlighted this issue. i.e.

INFO  [main] 2022-07-22 09:24:16,961 ColumnFamilyStore.java:2242 - Truncating system.size_estimates
INFO  [main] 2022-07-22 09:24:16,964 ColumnFamilyStore.java:2279 - Truncating system.size_estimates with truncatedAt=1658481856962
org.apache.cassandra.exceptions.TruncateException: Error during truncate: java.lang.IllegalArgumentException: Requested permits (0) must be positive

Using

docker container rm 59d4e2890a34

I removed the existing container and re-created it. The new version ran fine. I'm not sure what corrupted system.size_estimates.

Upvotes: 0

Jia Bochao
Jia Bochao

Reputation: 23

i found another solution

rm -rf apache-cassandra-4.0-beta4/data/saved_caches/*
rm -rf apache-cassandra-4.0-beta4/data/data/system/*

Upvotes: -2

Aaron
Aaron

Reputation: 57748

Is this a restart of a failed node? It looks like something is trying to replay a TRUNCATE command, but something required isn't being sent along (as per CASSANDRA-14905).

To get around this, I would remove the system_schema keyspace from the disk and restart. It should re-stream the schema from another node.

If that doesn't work, you might end up having to wipe the data and rebuild the node. And if that still doesn't work, you may have to remove/rejoin it to the cluster.

Edit:

As per Erick's answer, it looks like the problem was with snapshots directories for the size_estimates and table_esimates tables in the system keyspace. Remove those, and you should be good.

Upvotes: 1

Erick Ramirez
Erick Ramirez

Reputation: 16303

Question was also asked on the #cassandra channel on ASF Slack so I'm re-posting here for posterity.

Based on the stack trace, it looks like it's failing when trying to access the snapshot subdirectories of the system.size_estimates and system.table_estimates.

It's possible that the directories got corrupted as a result of the power outage (the reason for restarting the node in the first place) although I don't have sufficient information to confirm this.

In any case, the workaround is to manually delete the snapshot directories for the size_estimates/ and table_estimates/ in data/system/*. This will allow Cassandra to start successfully. Cheers!

P.S. 贾博超 confirmed on ASF Slack that the workaround allowed them to start C* on the node.

Upvotes: 2

Related Questions