Logback logs kept open by application and not deleting from disk

Question

I have an Spring-Boot application on Linux that writes to service logs using Logback. Logback is setup to write to a file (account-service.log) and rotate the file at 10.5MB and finally delete the file on the 8th rotate. This is the default configuration.

The issue I'm having is the first log file created is written to past the 10.5MB size limit (it does rotate at the 10.5MB mark). So it's increasing in size until it becomes the 8th file and then it tries to get deleted. At this point the file is 84MB (10.5MB * 8) size. You can see here the file increasing in size

The main problem with that is that the OS tries to delete the file but since the application, because of Logback, still keeps it open then the filesystem doesn't show the deleted file (like with command find or du) and the system still keeps the space allocated on the disk. Also the file is still written to so it's taking up more and more disk space. I ran sudo lsof | grep deleted to find out that this file isn't being fully deleted.

Also interestingly the group that that first file belongs to is different than the rest. The first file has group root while the rest have the correct group name of account-service. This might be why the application can't close the file but I'm not sure.

The Logback dependency comes from my Spring-Boot pom:

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.1.RELEASE</version>
</parent>
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>Dalston.SR4</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

The subsequent log files do get deleted ok and don't get written to past the 10.5 MB limit.

Has anyone seen this issue or found a way to fix it?

Answer 1

So I was able to solve most of my problems but one issue still remains.

The main issue was that I was using the built in logback xml for logging (base.xml). There it was configured for both console and file logging to be enabled.

<root level="INFO">
    <appender-ref ref="CONSOLE" />
    <appender-ref ref="FILE" />
</root>

So when I run it locally I get my logs written to my terminal and service log file which is good but the issue occurs on deployed instances. What happened was that both "CONSOLE" and "FILE" loggers where writing to the file which is why even after the first file rotated, it was still being written too. The "CONSOLE" logger was writing to the first file even though it rotated because from its point of view that file is the terminal. This also helped fix a double logging issue we were seeing.

My fix was to add my own logback xml (logback-spring.xml) to override the base.xml one.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<springProfile name="default">
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />
    <root level="INFO">
        <appender-ref ref="CONSOLE" />
    </root>
</springProfile>
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}/}spring.log}"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<root level="INFO">
    <appender-ref ref="FILE" />
</root>

This stops console logging on deployed instances and thus keeps the first file from logging past the 10.5MB limit. This fix fixed the disk space issue we experienced too.

The only issue I was not able to solve yet it why the file is kept open by the application and thus not allowing it to be fully deleted from the disk space.

Answer 2

As you have mentioned the file as root user access permissions which do not look correct. Can you check the application by doing ps -efw | grep applicationName to see what user permissions it has as your application should have application user permissions only.