CODEWITHSUNDEEP
spring-kafkaspring-cloud-stream
Alessandro Dionisi
Alessandro Dionisi

Reputation: 2684

Different key.serializer per producer when using transactions

I'm using spring-cloud-stream-kafka and I don't understand how is possible to change key.serializer property when using transactions. This is my configuration:

spring:
  cloud:
    stream:
      bindings:
        accountSource:
          contentType: application/*+avro
          destination: account
      kafka:
        binder:
          brokers: ${KAFKA_BOOTSTRAP_ADDRESSES}
          transaction:
            transaction-id-prefix: tx-
            producer:
              configuration:
                retries: 1
                acks: all                
        bindings:
          accountSource:
            producer:
              configuration:
                key.serializer: org.apache.kafka.common.serialization.StringSerializer # Ignored!
      schema:
        avro:
          subjectNamingStrategy: com.example.CustomSubjectNamingStrategy
      schemaRegistryClient:
        endpoint: ${KAFKA_SCHEMA_REGISTRY_URL}

I read here that when setting transaction properties at binder level, all other specific binding producer properties. Does this mean all producers in the applications should use the same key.serializer? It seems very restrictive to me.

Upvotes: 0

Views: 651

Answers (1)

Gary Russell
Gary Russell

Reputation: 174494

The problem is the transaction has to be started by the consumer binding (so it can send the offset to the transaction and commit or rollback after success/failure).

If there are multiple producer bindings, the consumer binding has no idea which one(s) the application will send data to, so we have to use one global producer.

One solution would be to write a custom delegating serializer and set a header in the output message to tell the serializer which delegate serializer to call.

Upvotes: 2

Related Questions