Samraj Subramaniam
Samraj Subramaniam

Reputation: 11

Distributed tracing in outbox

I have implemented outbox and working correctly. I am building connector image with the following Dockerfile

ARG DEBEZIUM_VERSION=2.1
FROM quay.io/debezium/connect:${DEBEZIUM_VERSION}
ENV KAFKA_CONNECT_JDBC_DIR=$KAFKA_CONNECT_PLUGINS_DIR/kafka-connect-jdbc \
    KAFKA_CONNECT_ES_DIR=$KAFKA_CONNECT_PLUGINS_DIR/kafka-connect-elasticsearch

ARG POSTGRES_VERSION=42.5.1
ARG KAFKA_JDBC_VERSION=5.3.2

# Deploy PostgreSQL JDBC Driver
RUN cd /kafka/libs && curl -sO https://jdbc.postgresql.org/download/postgresql-$POSTGRES_VERSION.jar

# Deploy Kafka Connect JDBC
RUN mkdir $KAFKA_CONNECT_JDBC_DIR && cd $KAFKA_CONNECT_JDBC_DIR &&\
    curl -sO https://packages.confluent.io/maven/io/confluent/kafka-connect-jdbc/$KAFKA_JDBC_VERSION/kafka-connect-jdbc-$KAFKA_JDBC_VERSION.jar

RUN mkdir /kafka/etc && cd /kafka/etc
COPY opentelemetry-javaagent.jar /kafka/etc

The image is started using following docker compose

connect:
  build:
    context: connector
  hostname: connect
  #image: quay.io/debezium/connect:2.5
  #image: sam-gmbh/connect:1.0
  depends_on:
    kafka-broker-1:
      condition: service_healthy
    kafka-broker-2:
      condition: service_healthy
    kafka-broker-3:
      condition: service_healthy
  ports:
    - 8084:8083
  environment:
    - BOOTSTRAP_SERVERS=kafka-broker-1:9092,kafka-broker-2:9092,kafka-broker-3:9092
    - ENABLE_APICURIO_CONVERTERS=true
    - GROUP_ID=1
    - CONFIG_STORAGE_TOPIC=my_connect_configs
    - OFFSET_STORAGE_TOPIC=my_connect_offsets
    - STATUS_STORAGE_TOPIC=my_connect_statuses
    - ENABLE_OTEL=true
    - KAFKA_OPTS=-javaagent:/kafka/etc/opentelemetry-javaagent.jar
    - OTEL_SERVICE_NAME=sam-gmbh-kafka-connect
    - OTEL_TRACES_EXPORTER=otlp
    - OTEL_METRICS_EXPORTER=none
    - OTEL_LOGS_EXPORTER=none
    - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318
    - OTEL_TRACES_SAMPLER=always_on
    - CONNECT_LOG4J_LOGGERS="DEBUG, stdout, appender"
    - OTEL_PROPAGATORS=tracecontext

While inserting an entry in outbox table, I have included a column called "tracingspancontext" and writing the tracing context in java.util.Properties in serialized format. For example, the content of the column "tracingspancontext"

#Wed Jun 26 16:29:09 CEST 2024
traceparent=00-a90cf8f35ba6dad17e8c9cdaafdbd0c1-3515e6f023a43bec-01

A connecter is deployed with the following script

{
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "tasks.max": "1",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres",
    "database.dbname" : "postgres",
    "table.include.list": "product.outbox, order.outbox, payment.outbox",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.route.by.field": "aggregate_type",
    "transforms.outbox.table.field.event.key": "aggregate_id",
    "transforms.outbox.table.field.event.payload": "payload",
    "transforms.outbox.table.expand.json.payload": "true",
    "transforms.outbox.route.topic.replacement": "${routedByValue}.events_ob",
    "transforms.outbox.tracing.span.context.field": "tracingspancontext",
    "transforms.outbox.tracing.operation.name": "debezium-read-sam",
    "transforms.outbox.tracing.with.context.field.only": "true",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "topic.prefix": "outbox",
    "value.converter.schemas.enable": false,
    "key.converter.schemas.enable": false,
    "slot.name": "outbox"
}

Unfortunately the tracing context is not propagated and connector starts a new trace. And adds a different tracing info in the header of the message than the one in the column "tracingspancontext".

Header from the message looks like below

{
    "traceparent": "00-d075554b98df047ba2795ca23e761969-77564be39bc4cc04-01",
    "id": "8bab4a69-976e-4842-8e5a-fce562847b38"
}

Can anyone help me how to propagate the context to connector and to the message?

I expect that tracing context written in the column is written the column "tracingspancontext" is propagated to connector and the message header contains the same tracing context.

Upvotes: 1

Views: 119

Answers (0)

Related Questions