CODEWITHSUNDEEP
javacassandrajvmruntimeexception
Piar
Piar

Reputation: 93

Cassandra - failure to start, with error "Insufficient disk space to write 572 bytes"

I am getting this error, trying to launch Cassandra. I ran rm -rf * on /var/log/cassandra and /var/lib/cassandra, and try to run cassandra again, with no success. Any idea? I have been looking to similar cases, but not at the launch of Cassandra, and nothing I found helped me to solve this problem.

root@test # cassandra
(...)
16:38:38.551 [MemtableFlushWriter:1] ERROR o.a.c.service.CassandraDaemon - Exception in thread Thread[MemtableFlushWriter:1,5,main]
java.lang.RuntimeException: Insufficient disk space to write 572 bytes
        at org.apache.cassandra.db.Directories.getWriteableLocation(Directories.java:349) ~[apache-cassandra-2.2.7.jar:2.2.7]
        at org.apache.cassandra.db.Memtable.flush(Memtable.java:324) ~[apache-cassandra-2.2.7.jar:2.2.7]
        at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1165) ~[apache-cassandra-2.2.7.jar:2.2.7]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_65]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
16:38:41.785 [ScheduledTasks:1] INFO  o.a.cassandra.locator.TokenMetadata - Updating topology for all endpoints that have changed

FYI, if I try to run cassandra again just after, I got 

[main] ERROR o.a.c.service.CassandraDaemon - Port already in use: 7199; nested exception is:
        java.net.BindException: Address already in use

So it seems that "CassandraDaemon" is alive; but if I want to run cqlsh I got this error: 

root@test # cqlsh
Warning: custom timestamp format specified in cqlshrc, but local timezone could not be detected.
Either install Python 'tzlocal' module for auto-detection or specify client timezone in your cqlshrc.

Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})

Finally, the free -m command gives me

              total        used        free      shared  buff/cache   available
Mem:          15758        8308         950        1440        6499        5576
Swap:          4095        2135        1960

Thanks for you help!

EDIT : Here are the WARN message I got during Cassandra launch:

11:36:14.622 [main] WARN  o.a.c.config.DatabaseDescriptor - Small commitlog volume detected at /var/lib/cassandra/commitlog; setting commitlog_total_space_in_mb to 2487.  You can override this in cassandra.yaml
11:36:14.623 [main] WARN  o.a.c.config.DatabaseDescriptor - Only 17 MB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
11:36:14.943 [main] WARN  o.a.cassandra.service.StartupChecks - jemalloc shared library could not be preloaded to speed up memory allocations
11:36:14.943 [main] WARN  o.a.cassandra.service.StartupChecks - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
11:36:14.943 [main] WARN  o.a.cassandra.service.StartupChecks - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
11:36:14.954 [main] WARN  o.a.cassandra.utils.SigarLibrary - Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : false, nproc limit adequate? : true

Upvotes: 0

Views: 2022

Answers (1)

Piar
Piar

Reputation: 93

As I said in a previous comment, it seems that the issue was triggered by the /var folder reaching its limit size, because Nifi, that I used for ingesting data, generates big logs in my configuration. Therefore removing useless logs and keeping an eye on the /var folder seems to prevent the error.

Upvotes: 1

Related Questions