How can I trace 100% of the requests done on my Spring Boot application?

Question

I have a Spring Boot 3 application that I want to monitor. I want to publish its traces to an OpenTelemetry collector so that they can be stored and displayed on a Grafana instance later.

Since this application is currently under testing, one of the requirements is that 100% of the traces be published to the collector to gather as much information as possible about the requests handled by it.

I created a brand new Spring Boot project from scratch that depends on spring-boot-starter-web and opentelemetry-spring-boot-starter.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.4.1</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>io.github.tiagoanleite</groupId>
    <artifactId>spring-boot-opentelemetry</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>spring-boot-opentelemetry</name>
    <description>Demo project for Spring Boot + OpenTelemetry</description>

    <properties>
        <java.version>17</java.version>
    </properties>
    
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.opentelemetry.instrumentation</groupId>
                <artifactId>opentelemetry-instrumentation-bom</artifactId>
                <version>2.11.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>


    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <!-- tag::actuator[] -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <!-- end::actuator[] -->
        
        <!-- tag::persistence[] -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jdbc</artifactId>
        </dependency>
        <dependency>
            <groupId>com.h2database</groupId>
            <artifactId>h2</artifactId>
        </dependency>
        <!-- end::persistence[] -->

        
        <!-- tag::opentelemetry[] -->
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry.contrib</groupId>
            <artifactId>opentelemetry-samplers</artifactId>
            <version>1.42.0-alpha</version>
        </dependency>
        <!-- end::opentelemetry[] -->

        <!-- tag::tests[] -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <!-- end::tests[] -->
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Based on Spring's tracing documentation, I also changed the tracing probability to 100% by adding the management.tracing.sampling.probability property to my application.yml:

spring:
  datasource:
    url: "jdbc:h2:mem:db"
management:
  endpoints:
    web:
      exposure:
        include: "*"
  tracing:
    sampling:
      probability: 1.0

To check if the application was publishing request traces, I created a simple REST controller that exposes a /hello endpoint:

package io.github.tiagoanleite;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class HelloController {

    @GetMapping("/hello")
    public String index() {
        return "Greetings from Spring Boot!";
    }

}

However not all the requests I do to my application result in a trace. From my experience, for each 10 GET /hello requests I get 3 or 4 traces published and not the 10 I was expecting.

What am I missing? This is my first time working with OpenTelemetry and I'm not sure if this is the inteded behavior or if there are any additional configurations I need to add to achieve my goal.

PS: If it helps, I have published the project mentioned in this question on my GitHub profile.

Answer 1

Your configuration is correct for the Spring Boot observability support, but your application is not using it. Your application is using the opentelemetry-spring-boot-starter which is not maintained by the Spring team. This means that it's not guaranteed that the observability-related Spring Boot docs apply to your case and you should instead refer to this starter documentation.

I believe you can use this resource attribute to configure that.

Note that this behavior difference is called out in the Spring Boot docs. If you meant to use the Spring Boot support, you can remove the io.opentelemetry dependencies, use the actuator starter and refer to the Spring Boot docs.