OpenTelemetry Dynatrace: SEVERE: Failed to export spans. The request could not be executed. Full error message: Canceled

Question

I have the following simple code:

package com.example;

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import io.opentelemetry.exporter.logging.LoggingSpanExporter;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;

public class ManualOpenTelemetryExample {

    public static void main(String[] args) {
        // Step 1: Set up the OTLP exporter to send data to Dynatrace
        OtlpGrpcSpanExporter otlpExporter = OtlpGrpcSpanExporter.builder()
                .setEndpoint("<YOUR_DYNATRACE_ENDPOINT>") // Replace with your Dynatrace endpoint
                .addHeader("Authorization", "Api-Token <YOUR_API_TOKEN>") // Replace with your Dynatrace API token
                .build();

        // Step 2: Set up the LoggingSpanExporter for debugging
        LoggingSpanExporter loggingExporter = LoggingSpanExporter.create();

        // Step 3: Set up the BatchSpanProcessor with the OTLP exporter
        BatchSpanProcessor spanProcessor = BatchSpanProcessor.builder(otlpExporter).build();

        // Step 4: Set up the TracerProvider with both processors
        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
                .addSpanProcessor(spanProcessor)
                .addSpanProcessor(SimpleSpanProcessor.create(loggingExporter)) // Add logging exporter
                .build();

        // Step 5: Initialize OpenTelemetry with the TracerProvider
        OpenTelemetrySdk openTelemetry = OpenTelemetrySdk.builder()
                .setTracerProvider(tracerProvider)
                .build();

        // Step 6: Get a Tracer
        Tracer tracer = openTelemetry.getTracer("manual-instrumentation-example", "1.0.0");

        // Step 7: Simulate a workflow with multiple spans
        simulateWork(tracer);

        // Step 8: Shutdown the tracer provider to flush spans
        tracerProvider.shutdown();
    }

    /**
     * Simulates a workflow with multiple spans.
     */
    private static void simulateWork(Tracer tracer) {
        // Create a parent span for the entire workflow
        Span parentSpan = tracer.spanBuilder("workflow-operation").startSpan();
        try (Scope parentScope = parentSpan.makeCurrent()) {
            // Add attributes to the parent span
            parentSpan.setAttribute("service.name", "my-java-service");
            parentSpan.setAttribute("environment", "production");

            // Simulate step 1: Fetch data
            simulateStep(tracer, "fetch-data", 200);

            // Simulate step 2: Process data
            simulateStep(tracer, "process-data", 300);

            // Simulate step 3: Save data
            simulateStep(tracer, "save-data", 100);

            // Simulate step 4: Send notification
            simulateStep(tracer, "send-notification", 150);

        } finally {
            // End the parent span
            parentSpan.end();
        }
    }

    /**
     * Simulates a single step in the workflow.
     *
     * @param tracer       The OpenTelemetry tracer.
     * @param stepName     The name of the step.
     * @param durationMs   The duration of the step in milliseconds.
     */
    private static void simulateStep(Tracer tracer, String stepName, long durationMs) {
        // Create a span for the step
        Span stepSpan = tracer.spanBuilder(stepName).startSpan();
        try (Scope stepScope = stepSpan.makeCurrent()) {
            // Add attributes to the step span
            stepSpan.setAttribute("step.name", stepName);
            stepSpan.setAttribute("step.duration.ms", durationMs);

            // Simulate work by sleeping
            System.out.println("Executing step: " + stepName);
            Thread.sleep(durationMs);

        } catch (InterruptedException e) {
            e.printStackTrace();
        } finally {
            // End the step span
            stepSpan.end();
        }
    }
}

All I want to do is to create one simple java process just to test creating ingest span into Dynatrace. But I'm always ended with the following error:

$ mvn exec:java -Dexec.mainClass="com.example.ManualOpenTelemetryExample"
[INFO] Scanning for projects...
[INFO]
[INFO] --------------< com.example:manual-opentelemetry-example >--------------
[INFO] Building manual-opentelemetry-example 1.0-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- exec:3.5.0:java (default-cli) @ manual-opentelemetry-example ---
Executing step: fetch-data
Feb 03, 2025 3:11:25 PM io.opentelemetry.exporter.logging.LoggingSpanExporter export
INFO: 'fetch-data' : b6176f27f93041eda5e3ea2c20e27870 7942d9ee73773df1 INTERNAL [tracer: manual-instrumentation-example:1.0.0] AttributesMap{data={step.duration.ms=200, step.name=fetch-data}, capacity=128, totalAddedValues=2}
Executing step: process-data
Feb 03, 2025 3:11:25 PM io.opentelemetry.exporter.logging.LoggingSpanExporter export
INFO: 'process-data' : b6176f27f93041eda5e3ea2c20e27870 d3b9bc007fb9e028 INTERNAL [tracer: manual-instrumentation-example:1.0.0] AttributesMap{data={step.duration.ms=300, step.name=process-data}, capacity=128, totalAddedValues=2}
Executing step: save-data
Feb 03, 2025 3:11:26 PM io.opentelemetry.exporter.logging.LoggingSpanExporter export
INFO: 'save-data' : b6176f27f93041eda5e3ea2c20e27870 778ede14f1ca0d99 INTERNAL [tracer: manual-instrumentation-example:1.0.0] AttributesMap{data={step.duration.ms=100, step.name=save-data}, capacity=128, totalAddedValues=2}
Executing step: send-notification
Feb 03, 2025 3:11:26 PM io.opentelemetry.exporter.logging.LoggingSpanExporter export
INFO: 'send-notification' : b6176f27f93041eda5e3ea2c20e27870 17ce27eecaee9ac4 INTERNAL [tracer: manual-instrumentation-example:1.0.0] AttributesMap{data={step.duration.ms=150, step.name=send-notification}, capacity=128, totalAddedValues=2}
Feb 03, 2025 3:11:26 PM io.opentelemetry.exporter.logging.LoggingSpanExporter export
INFO: 'workflow-operation' : b6176f27f93041eda5e3ea2c20e27870 e350cdb78679c4e7 INTERNAL [tracer: manual-instrumentation-example:1.0.0] AttributesMap{data={environment=production, service.name=my-java-service}, capacity=128, totalAddedValues=2}
Feb 03, 2025 3:11:26 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
SEVERE: Failed to export spans. The request could not be executed. Full error message: Canceled
[WARNING] thread Thread[#55,Okio Watchdog,5,com.example.ManualOpenTelemetryExample] was interrupted but is still alive after waiting at least 14980msecs
[WARNING] thread Thread[#55,Okio Watchdog,5,com.example.ManualOpenTelemetryExample] will linger despite being asked to die via interruption
[WARNING] NOTE: 1 thread(s) did not finish despite being asked to via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16.700 s
[INFO] Finished at: 2025-02-03T15:11:41+08:00
[INFO] ------------------------------------------------------------------------

What could be wrong here? Is there anything missing from my code?

The idea is to test my Dynatrace instance that opentelemetry is proven to be working using Java

Answer 1

It seems like you are trying to use the OtlpGrpcSpanExporter. gRPC is currently not supported. Could you try swapping out the OtlpGrpcSpanExporter for an OtlpHttpSpanExporter? This would mean data is exported via OTLP HTTP to the Dynatrace endpoint.