Glassfish4 log rotation "Maximum History Files" issue

Question

Environment: Glassfish 4.0 (only one DAS), Windows Server 2012 R2, Java 1.7.0_51 Create the DAS instance service by using the create-service subcommand.

Issue: The maximum history files attribute has been set, however, Glassfish Server couldn’t remove the old log files due to the lock file server.log.lck

Path --> C:\glassfish4\glassfish\domains\domain1\config\logging.properties com.sun.enterprise.server.logging.GFFileHandler.maxHistoryFiles=10

Log Snippet:

[2014-12-10T18:00:39.372+0900] [glassfish 4.0] [SEVERE] [] [] [tid: _ThreadID=16 _ThreadName=Thread-5] [timeMillis: 1418202039372] [levelValue: 1000] [[
  java.util.logging.ErrorManager: 0: FATAL ERROR: COULD NOT DELETE LOG FILE.]]

[2014-12-10T18:00:39.372+0900] [glassfish 4.0] [SEVERE] [] [] [tid: _ThreadID=16 _ThreadName=Thread-5] [timeMillis: 1418202039372] [levelValue: 1000] [[
  java.io.IOException: Could not delete log file: C:\glassfish4\glassfish\domains\domain1\logs\server.log.lck
    at com.sun.enterprise.server.logging.GFFileHandler.cleanUpHistoryLogFiles(GFFileHandler.java:725)
    at com.sun.enterprise.server.logging.GFFileHandler$4.run(GFFileHandler.java:802)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.sun.enterprise.server.logging.GFFileHandler.rotate(GFFileHandler.java:744)
    at com.sun.enterprise.server.logging.GFFileHandler$1.run(GFFileHandler.java:301)
    at com.sun.enterprise.server.logging.LogRotationTimerTask.run(LogRotationTimerTask.java:68)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)]]

Findings:

1, If the lock file “server.log.lck” exists in the log folder, the issue occurred, and can find the above errors in the log every day when Glassfish server tries to remove the old log files. If there is no “server.log.lck” in the log folder, no any issue and work properly.

2, If the DAS instance is started by the command “asadmin start-domain domain1”, there is no lock file “server.log.lck” generated in the log folder. But if the DAS instance is started in Windows Service, the lock file “server.log.lck” will be generated automatically and keep 0KB until stop the service, this file will be removed automatically.

3, If the DAS instance is started by the command “asadmin start-domain -w domain1” which adds the watchdog option, the lock file “server.log.lck” will be generated automatically and exist until stop the service.

4, When the lock file “server.log.lck” appears, there is always one more java.exe process existing. Therefore, when start the DAS instance from Windows Service, there are two “java.exe” running in the process and “server.log.lck” is using by one of them.

Questions:

1, I’d like to start/stop the DAS instance by Windows Service, not using the subcommand. Moreover, I don’t want to keep all Glassfish logs on my server and it will cause a disk full issue so that I would prefer to turn on the Glassfish Logging Maximum history Files option. Is there any workaround or solution for that?

2, Is this a defect of Glassfish or it’s just a setting issue? I did try to install on other servers and all had the same issue.

3, Why there are two java.exe processes running if started from Windows Server, is the 2nd one used for “watchdog”?

Thank you so much for your help and please let me know if there is any further information you’d like to know or want me to do some other tests.

Answer 1

In case someone is still struggling I found a solution. When you create a GF service in Windows environment via asadmin create-service GF creates a file domain1Service.xml in glassfish\domains\domain1\bin which contains parametres for server to start. It looks something like the following

    <service>
      <id>domain1</id>
      <name>domain1 GlassFish Server</name>
      <description>GlassFish Server</description>
      <executable>C:/project/glassfish-4.1/glassfish/lib/nadmin.bat</executable>
      <logpath>C:\\project\\glassfish-4.1\\glassfish\\domains/domain1/bin</logpath>
      <logmode>reset</logmode>
      <depend>tcpip</depend>
      <startargument>start-domain</startargument>
          <startargument>--watchdog</startargument>
      <startargument>--domaindir</startargument>
          <startargument>C:\\project\\glassfish-4.1\\glassfish\\domains</startargument>
      <startargument>domain1</startargument>
          <stopargument>stop-domain</stopargument>
      <stopargument>--domaindir</stopargument>
          <stopargument>C:\\project\\glassfish-4.1\\glassfish\\domains</stopargument>
      <stopargument>domain1</stopargument>
    </service>         

the line <startargument>--watchdog</startargument> is responsible for launching watchdog process which prevents log file from being deleted. You can't just delete this startargument section (the service won't start) but you can switch this off by setting false flag like this

<startargument>--watchdog=false</startargument>

After that the service will start like via manual start-domain command without watchdog process. You should do it after every service creation and it could be pretty annoying so I did further research. It turns out that asadmin create OS specific domainService.xml by using templates located in glassfish\lib\install\templates Those templates also OS specific. And template for Windows (named Domain-service-winsw.xml.template) looks like this

    <service>
      <id>%%%NAME%%%</id>
      <name>%%%DISPLAY_NAME%%%</name>
      <description>GlassFish Server</description>
      <executable>%%%AS_ADMIN_PATH%%%</executable>
      <logpath>%%%LOCATION%%%/%%%ENTITY_NAME%%%/bin</logpath>
      <logmode>reset</logmode>
      <depend>tcpip</depend>
      <startargument>%%%START_COMMAND%%%</startargument>
      <startargument>--watchdog</startargument>
      %%%CREDENTIALS_START%%%%%%LOCATION_ARGS_START%%%<startargument>%%%ENTITY_NAME%%%</startargument>
      <stopargument>%%%STOP_COMMAND%%%</stopargument>
      %%%CREDENTIALS_STOP%%%%%%LOCATION_ARGS_STOP%%%<stopargument>%%%ENTITY_NAME%%%</stopargument>
    </service>

So you can edit template directly by setting param --watchdog=false and this change will reflect in all future created files domainService.xml

Hope it helps.

Answer 2

That’s not the right solution. Watchdog has an important function: it monitors whether the service is running or not. Without watchdog, glassfish is started correctly, but shortly afterwards the system no longer knows if the service is still running or maybe crashed. In the Services GUI, only the “start” button is active (always!). A “stop” and “restart” cannot be used. A right solution would be the possibility to change the path to the lock file.