CODEWITHSUNDEEP
javaspring-bootliquibase
mihristov
mihristov

Reputation: 356

Liquibase/Springboot startup exception

Starting yesterday (Sunday) morning my production app fails to start, with no code changes from my side. It's running Springboot 2.3.4, Liquibase-core 3.8.0 and is hosted on Amazon linux2. Funny thing is there are no exceptions locally, only when deployed.

Here is the relevant stack trace:

Caused by: liquibase.exception.UnexpectedLiquibaseException: java.nio.file.NoSuchFileException: /tmp/agent12302722365010540729.jar
 at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:129)
 at liquibase.servicelocator.ServiceLocator.<init>(ServiceLocator.java:69)
 at liquibase.servicelocator.CustomResolverServiceLocator.<init>(CustomResolverServiceLocator.java:16)
 at org.springframework.boot.liquibase.LiquibaseServiceLocatorApplicationListener$LiquibasePresent.replaceServiceLocator(LiquibaseServiceLocatorApplicationListener.java:55)
 at org.springframework.boot.liquibase.LiquibaseServiceLocatorApplicationListener.onApplicationEvent(LiquibaseServiceLocatorApplicationListener.java:44)
 at org.springframework.boot.liquibase.LiquibaseServiceLocatorApplicationListener.onApplicationEvent(LiquibaseServiceLocatorApplicationListener.java:36)
...
Caused by: java.nio.file.NoSuchFileException: /tmp/agent801508645517312012.jar
  at liquibase.resource.ClassLoaderResourceAccessor.getResourcesAsStream(ClassLoaderResourceAccessor.java:53)
  at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:115)

I double checked all application related files and env variables and they are all the same. The file in question is in no way related to my app.

Do you have any idea what this file is and why is Liquibase trying to find it all of the sudden?

Upvotes: 14

Views: 2187

Answers (4)

Pavla
Pavla

Reputation: 155

To elaborate a bit more on what's happening here:

AWS has developed a systemd service that, every second, looks for any JVMs running on the system. For all that it finds and supports it connect a java agent that at runtime replaces the log4j JNDI lookup() method with a hardcoded safe string.

Why liquibase fails on this is not clear to me. I do see that the already given answers seem to be too invasive. There's no need to remove/reinstall/update your JVM. You can work around this by:

  • Stopping the systemd patch service: sudo service log4j-cve-2021-44228-hotpatch status|stop|start

  • Best solution - create the kill file that, if read by the service, stops applying the patch: sudo touch /etc/log4j-cve-2021-44228-hotpatch.kill

Upvotes: 4

Karl Kild&#233;n
Karl Kild&#233;n

Reputation: 2435

For us moving to latest liquibase (4+) resolved this. They use a more standard ServiceLocator in 4+ and that seems to make the difference.

Upvotes: 3

Reda
Reda

Reputation: 1335

I had the same problem. On the startup of the amazon linux 2, there is a security patch that is installed.

The package causing the problem is log4j-cve-2021-44228-hotpatch.noarch (you can check that in /var/log/yum.log)

A temporary solution is to uninstall the patch and install another java version.

yum remove log4j-cve-2021-44228-hotpatch.noarch
yum install java-11-openjdk-11.0.12.0.7-0.amzn2.0.2.x86_64

Thanks to @mihristov for the solution.

Upvotes: 6

tiktock
tiktock

Reputation: 991

This is a temporary issue with: java-11-amazon-corretto-headless-11.0.13+8-2.amzn2.aarch64

To fix:

yum remove log4j-cve-2021-44228-hotpatch.noarch
yum install java-11-amazon-corretto-headless-11.0.13+8-1.amzn2.aarch64

Thanks to @Reda and @mihristov for pointing the issue out.

Our actual fix was just to install a specific version of corretto instead of the general package

yum install java-11-amazon-corretto-headless-11.0.13+8-1.amzn2.aarch64

You should only do this temporarily until a fix is out!

Upvotes: 2

Related Questions