Reputation: 11
I have implemented outbox and working correctly. I am building connector image with the following Dockerfile
ARG DEBEZIUM_VERSION=2.1
FROM quay.io/debezium/connect:${DEBEZIUM_VERSION}
ENV KAFKA_CONNECT_JDBC_DIR=$KAFKA_CONNECT_PLUGINS_DIR/kafka-connect-jdbc \
KAFKA_CONNECT_ES_DIR=$KAFKA_CONNECT_PLUGINS_DIR/kafka-connect-elasticsearch
ARG POSTGRES_VERSION=42.5.1
ARG KAFKA_JDBC_VERSION=5.3.2
# Deploy PostgreSQL JDBC Driver
RUN cd /kafka/libs && curl -sO https://jdbc.postgresql.org/download/postgresql-$POSTGRES_VERSION.jar
# Deploy Kafka Connect JDBC
RUN mkdir $KAFKA_CONNECT_JDBC_DIR && cd $KAFKA_CONNECT_JDBC_DIR &&\
curl -sO https://packages.confluent.io/maven/io/confluent/kafka-connect-jdbc/$KAFKA_JDBC_VERSION/kafka-connect-jdbc-$KAFKA_JDBC_VERSION.jar
RUN mkdir /kafka/etc && cd /kafka/etc
COPY opentelemetry-javaagent.jar /kafka/etc
The image is started using following docker compose
connect:
build:
context: connector
hostname: connect
#image: quay.io/debezium/connect:2.5
#image: sam-gmbh/connect:1.0
depends_on:
kafka-broker-1:
condition: service_healthy
kafka-broker-2:
condition: service_healthy
kafka-broker-3:
condition: service_healthy
ports:
- 8084:8083
environment:
- BOOTSTRAP_SERVERS=kafka-broker-1:9092,kafka-broker-2:9092,kafka-broker-3:9092
- ENABLE_APICURIO_CONVERTERS=true
- GROUP_ID=1
- CONFIG_STORAGE_TOPIC=my_connect_configs
- OFFSET_STORAGE_TOPIC=my_connect_offsets
- STATUS_STORAGE_TOPIC=my_connect_statuses
- ENABLE_OTEL=true
- KAFKA_OPTS=-javaagent:/kafka/etc/opentelemetry-javaagent.jar
- OTEL_SERVICE_NAME=sam-gmbh-kafka-connect
- OTEL_TRACES_EXPORTER=otlp
- OTEL_METRICS_EXPORTER=none
- OTEL_LOGS_EXPORTER=none
- OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318
- OTEL_TRACES_SAMPLER=always_on
- CONNECT_LOG4J_LOGGERS="DEBUG, stdout, appender"
- OTEL_PROPAGATORS=tracecontext
While inserting an entry in outbox table, I have included a column called "tracingspancontext" and writing the tracing context in java.util.Properties in serialized format. For example, the content of the column "tracingspancontext"
#Wed Jun 26 16:29:09 CEST 2024
traceparent=00-a90cf8f35ba6dad17e8c9cdaafdbd0c1-3515e6f023a43bec-01
A connecter is deployed with the following script
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "postgres",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgres",
"database.dbname" : "postgres",
"table.include.list": "product.outbox, order.outbox, payment.outbox",
"transforms": "outbox",
"transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
"transforms.outbox.route.by.field": "aggregate_type",
"transforms.outbox.table.field.event.key": "aggregate_id",
"transforms.outbox.table.field.event.payload": "payload",
"transforms.outbox.table.expand.json.payload": "true",
"transforms.outbox.route.topic.replacement": "${routedByValue}.events_ob",
"transforms.outbox.tracing.span.context.field": "tracingspancontext",
"transforms.outbox.tracing.operation.name": "debezium-read-sam",
"transforms.outbox.tracing.with.context.field.only": "true",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"topic.prefix": "outbox",
"value.converter.schemas.enable": false,
"key.converter.schemas.enable": false,
"slot.name": "outbox"
}
Unfortunately the tracing context is not propagated and connector starts a new trace. And adds a different tracing info in the header of the message than the one in the column "tracingspancontext".
Header from the message looks like below
{
"traceparent": "00-d075554b98df047ba2795ca23e761969-77564be39bc4cc04-01",
"id": "8bab4a69-976e-4842-8e5a-fce562847b38"
}
Can anyone help me how to propagate the context to connector and to the message?
I expect that tracing context written in the column is written the column "tracingspancontext" is propagated to connector and the message header contains the same tracing context.
Upvotes: 1
Views: 119