org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch

Question

This was working without any problems since months, but these days, I am getting this error, any help would be greatly appreciated.

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /localhost:9042 (com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on /localhost:9042: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 50809440-5f13-11e6-9601-f72eff140960; expected 507fd0f0-5f13-11e6-aa2f-955d215422cf)), /localhost:9042 (com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on /localhost:9042: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 50809440-5f13-11e6-9601-f72eff140960; expected 507fd0f0-5f13-11e6-aa2f-955d215422cf)), /localhost:9042 (com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on /localhost:9042: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 50809440-5f13-11e6-9601-f72eff140960; expected 507fd0f0-5f13-11e6-aa2f-955d215422cf))) com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /localhost:9042 (com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on /localhost:9042: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 50809440-5f13-11e6-9601-f72eff140960; expected 507fd0f0-5f13-11e6-aa2f-955d215422cf)), /localhost:9042 (com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on /localhost:9042: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 50809440-5f13-11e6-9601-f72eff140960; expected 507fd0f0-5f13-11e6-aa2f-955d215422cf)), /localhost:9042 (com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on /localhost:9042: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 50809440-5f13-11e6-9601-f72eff140960; expected 507fd0f0-5f13-11e6-aa2f-955d215422cf)))

Even while dropping the already created table using DataStax DevCenter, I see the following :

Answer 1

1. On each node check file system and see if you have two directories for the table with cf-id 50809440-5f13-11e6-9601-f72eff140960 and 507fd0f0-5f13-11e6-aa2f-955d215422cf in data directory

2. If you have two directories. RUN below command to get latest cf_id.

   cqlsh -e "select cf_id from system.schema_columnfamilies where columnfamily_name = 'YOUR_CF_NAME' allow filtering"

3. Move data from new cf_id to older one and remove old folders.

4. Run nodetool refresh

This should resolve problem. If not then you should do a rolling restart.

Answer 2

You can try nodetool describecluster to identify any differences in schema and nodetool resetlocalschema on any nodes that are off on their own. Them getting off may mean something like significant drift in clocks (setup NTP) but it really depends on version. When updating schema try to wait for agreement before making more changes.