CODEWITHSUNDEEP
cassandracassandra-3.0
developthou
developthou

Reputation: 363

Cassandra old hint files present in hints folder

During cassandra decommission I noticed that the node tries to send hints which takes extremely long time and never completes. I checked the hints folder and found hints that are more than 9+ months old. I am not sure why those old hints are still present in the folder so I decided to delete them. After I deleted them I noticed the following entry in the system.log

INFO  [HintsDispatcher:1070] 2021-07-08 11:32:01,056  HintsDispatchExecutor.java:141 - Transferring all hints to /10.199.190.233: 7935f1b5-4725-4dc2-ad6d-b883d53d907d
ERROR [HintsDispatcher:1070] 2021-07-08 11:32:01,061  CassandraDaemon.java:207 - Exception in thread Thread[HintsDispatcher:1070,1,RMI Runtime]
java.lang.RuntimeException: java.nio.file.NoSuchFileException: /data/cassandra/data/hints/ce6bb0e3-849f-487d-9274-38b8536b89cf-1603947885707-1.hints

Where does Cassandra keep metadata for the hints as system.hints folder didn't have any entry?

Cassandra Version is 3.0.12

Upvotes: 2

Views: 1359

Answers (2)

Zeitounator
Zeitounator

Reputation: 44615

Answer from @Erick Ramirez is perfectly valid. The downside is that it removes all hints.

If you just want to remove leftover hints (as in the question) which are way past the max_hint_window_in_ms (which defaults to 3 hours), or even if you ran into an error because of a corrupted hint, here is a way to remove only selected hints safely inspired by the datastax documentation:

  1. Turn cassandra off on the given node, make sure there are no leftover java processes
  2. Delete the selected hints from the hint filesystem (make sure to delete both the .hints and .crc32 files if they are present)
  3. Start cassandra.

Upvotes: 0

Erick Ramirez
Erick Ramirez

Reputation: 16303

There is a catalog of hints held in memory on each Cassandra node for tracking.

If you manually delete the contents of the hints directory on a node, the entries in the hints catalog become stale and you run into the NoSuchFileException you posted.

The correct way of deleting hints is with the nodetool truncatehints command. Cheers!

Upvotes: 2

Related Questions