CODEWITHSUNDEEP
javaspring-bootprometheushystrixmicrometer
Matteo Grimaldi
Matteo Grimaldi

Reputation: 61

Can't expose hystrix metrics to /actuator/prometheus after migrating application to Spring boot 2

After migrating my application (stack spring boot, camel, hystrix) to spring boot 2 from spring boot 1.5. I can't get the hystrix metrics to show up in /actuator/prometheus.

As many solutions and tutorials suggest, I've made sure I had the following dependencies

<dependency>
   <groupId>io.micrometer</groupId>
   <artifactId>micrometer-registry-prometheus</artifactId>
   <version>${micrometer-version}</version>
</dependency>
<dependency>
   <groupId>io.micrometer</groupId>
   <artifactId>micrometer-core</artifactId>
   <version>${micrometer-version}</version>
</dependency>

And added the following configuration class, which I'm sure it's being instantiated, as I've checked in spring boot autoconfiguration logs:

import io.micrometer.core.instrument.binder.hystrix.HystrixMetricsBinder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class HystrixMetricsConfig {

   @Bean
   HystrixMetricsBinder registerHystrixMetricsBinder() {
      return new HystrixMetricsBinder();
   }
}

To avoid assuming such tutorials/answers were correct by definition, I've set up a simple spring boot project from scratch with a simple controller, hystrix and the above dependencies, and metrics do appear at /actuator/prometheus, without additional steps.

My application has more dependencies that the simple project, so I think something else might be overriding/disabling the metrics binding.

[...]
<dependencies>

   <!-- Spring Boot -->
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
   </dependency>
   <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
      <exclusions>
         <exclusion>
            <groupId>om.netflix.archaius</groupId>
            <artifactId>archaius-core</artifactId>
         </exclusion>
      </exclusions>
   </dependency>
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
   </dependency>
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-thymeleaf</artifactId>
   </dependency>     
   <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-config-server</artifactId>
   </dependency>
   <dependency>
      <groupId>org.springframework.security</groupId>
      <artifactId>spring-security-web</artifactId>
   </dependency>

   <dependency>
      <groupId>org.jolokia</groupId>
      <artifactId>jolokia-core</artifactId>
   </dependency>



   <!-- Camel -->
   <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-spring-boot-starter</artifactId>
   </dependency>
   <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-spring-security</artifactId>
  </dependency>
   <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-swagger-java-starter</artifactId>
   </dependency>
   <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-servlet-starter</artifactId>
   </dependency>
   <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-http4</artifactId>
   </dependency>
   <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-spring</artifactId>
   </dependency>
   <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-hystrix-starter</artifactId>
   </dependency>
   <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-metrics</artifactId>
   </dependency>     
   <!-- Spring Framework Caching Support -->
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-cache</artifactId>
   </dependency>
   <dependency>
      <groupId>com.hazelcast</groupId>
      <artifactId>hazelcast</artifactId>
   </dependency>
   <dependency>
      <groupId>com.hazelcast</groupId>
      <artifactId>hazelcast-spring</artifactId>
   </dependency>
   <!-- logging -->
   <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
   </dependency>
   <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-core</artifactId>
   </dependency>
   <dependency>
      <groupId>net.logstash.logback</groupId>
      <artifactId>logstash-logback-encoder</artifactId>
      <scope>compile</scope>
   </dependency>
   <dependency>
       <groupId>org.apache.ws.security</groupId>
       <artifactId>wss4j</artifactId>
   </dependency>     
   <dependency>
       <groupId>com.google.guava</groupId>
       <artifactId>guava</artifactId>
   </dependency>         
   <!-- Test -->
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
   </dependency>
   <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-test</artifactId>
      <scope>test</scope>
   </dependency>
   <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <scope>provided</scope>
   </dependency>
   <!-- stream -->       
   <dependency>
      <groupId>com.netflix.archaius</groupId>
      <artifactId>archaius-core</artifactId>
      <version>0.7.6</version>
   </dependency>
   <dependency>
      <groupId>io.micrometer</groupId>
     <artifactId>micrometer-registry-prometheus</artifactId>
   </dependency>
   <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-core</artifactId>
   </dependency>
   <dependency>
      <groupId>org.reactivestreams</groupId>
      <artifactId>reactive-streams</artifactId>
      <version>1.0.2</version>
   </dependency>
</dependencies>
[...]

More than being told what the actual problem is, I'm struggling trying to understand how to troubleshoot such issues. Is there any micrometer/Spring boot log (other than autoconfiguration report) that could be activated to understand what's going on ?

Upvotes: 3

Views: 1634

Answers (1)

Matteo Grimaldi
Matteo Grimaldi

Reputation: 61

After having followed the whole custom hystrix metrics binding procedure, I've ended up discovering everything was getting bound correctly, at least in principle.

I've then tried to trigger a breakpoint at the first execution of an hystrix command to check what was going on with the registry and the publisher. Placing a breakpoint in the method

HystrixMetricsPublisherThreadPool getPublisherForThreadPool(HystrixThreadPoolKey threadPoolKey, HystrixThreadPoolMetrics metrics, HystrixThreadPoolProperties properties)

of HystrixMetricsPublisherFactory class, I found out that the HystrixPlugins instance was different with respect to the the moment in which the publisher was set. After inspecting all the code of the application, I found out that, to set a custom event notifier, the instance was getting reset.

@EventListener(ApplicationReadyEvent.class)
    public void doAfterStartup() {
        Hystrix.reset();
        registerCustomHystrixEventNotifier(circuitBreakerHystrixEventNotifier);
        logger.info("hello world, application started up");
    }

I've then modified the method in order to register the event notifier, without resetting what the autoconfiguration configured before, and hystrix metrics are now showing up in the prometheus endpoint.

Upvotes: 1

Related Questions