Monitoring custom Stream apps in Spring Cloud data flow

Question

I am trying scdf and its monitoring with prometheus and grafana. I followed the documentation available and able to deploy the sample stream and able to see the metrics in the grafana.

I have created a stream with some custom stream app (other than the supplied rabbit mq starter apps).

Stream: htt | participant | log

But am not able see the "participant" application metrics in gafana. But able to see the metrics of http and log apps.

Added below properties in application.properties.

management.endpoint.metrics.enabled=true
management.endpoints.web.exposure.include=*
management.endpoint.prometheus.enabled=true
management.metrics.export.prometheus.enabled=true
spring.cloud.streamapp.security.enabled=false

Added below dependencies:

<dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <!--&lt;!&ndash; https://mvnrepository.com/artifact/org.springframework.cloud.stream.app/app-starters-common &ndash;&gt;-->
    <!--<dependency>-->
        <!--<groupId>org.springframework.cloud.stream.app</groupId>-->
        <!--<artifactId>app-starters-common</artifactId>-->
        <!--<version>2.1.1.RELEASE</version>-->
        <!--<type>pom</type>-->
    <!--</dependency>-->

    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-amqp</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-stream</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-stream-binder-rabbit</artifactId>
    </dependency>

After adding app-starters-common:org.springframework.cloud.stream.app dependency localhost:< port >/ opens a login page.

Answer 1

I came across this question while looking answers for the same.

Here is my working code snippet:-

Pom.xml

    <dependency>
            <groupId>org.springframework.cloud.stream.app</groupId>
            <artifactId>app-starters-micrometer-common</artifactId>
            <version>2.1.2.RELEASE</version>
    </dependency>
    <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>
    <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-jmx</artifactId>
    </dependency>

application property change to enable prometheus and disable security(login page)

management.endpoints.web.exposure.include=*
management.metrics.export.prometheus.enabled=true

-- this one to remove security (login page), which was automatically added by app-starter-micrometre-common dependency.

spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration, org.springframework.boot.actuate.autoconfigure.security.servlet.ManagementWebSecurityAutoConfiguration

or

exclude the dependency, I excluded config-client and few other since I don't need them in my application.

<dependency>
            <groupId>org.springframework.cloud.stream.app</groupId>
            <artifactId>app-starters-micrometer-common</artifactId>
            <version>2.1.2.RELEASE</version>
            <exclusions>
                <exclusion>
                    <artifactId>spring-security-config</artifactId>
                    <groupId>org.springframework.security</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>spring-cloud-services-starter-config-client</artifactId>
                    <groupId>io.pivotal.spring.cloud</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>*</artifactId>
                    <groupId>org.springframework.boot</groupId>
                </exclusion>
            </exclusions>
        </dependency>

Answer 2

With later Data Flow 2.3.x, you need to add the following dependencies to your processor:

<dependency>
    <groupId>org.springframework.cloud.stream.app</groupId>
    <artifactId>app-starters-micrometer-common</artifactId>
    <version>2.1.2.RELEASE</version>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>    
<dependency>
    <groupId>io.micrometer.prometheus</groupId>
    <artifactId>prometheus-rsocket-spring</artifactId>
    <version>0.9.0</version>
</dependency>

The app-starters-micrometer-common injects DataFlow specific tags, such as stream.name, application.name, application.type all used by the dashboard to aggregate the required metrics.

In addition you can follow instructions in the sample projects, showing how to build custom Source, Processor and Sink apps with enabled prometheus monitoring: https://github.com/spring-cloud/spring-cloud-dataflow-samples/tree/master/monitoring-samples/stream-apps

Answer 3

I think you need app-starters-micrometer-common dependency which adds some of the micrometer tags to your app. This dependency is intended to be used by the Spring cloud stream app starters and I believe you can use it in your custom application as well.