CODEWITHSUNDEEP
c#apache-kafka
101000101
101000101

Reputation: 33

Does Kafka allow reading a message content asynchronously?

Anybody knows whether the Kafka clients allows sending and reading the content of a message in an asynchronous way.

I am currently using the Confluent.Kafka producers and consumers in C# which allows making an async call containing the whole message payload, however it would be interesting to publish the value of the message or content of several MBs asynchronously, and being able of reading it asynchronously as well, instead of just receiving the message in one shot.

using (var producer = new ProducerBuilder<string, string>(config).Build())
{
     await producer.ProduceAsync(_topic, new Message<string, string> { Key = _file, Value = <pass async content here> });
}

Is anyway of achieving this?

Thanks

Upvotes: 1

Views: 905

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191854

The producer needs to flush the event, send to the broker, which gets written to disk and (optionally) ack the entire record before consumers can read it.

If you'd like to stream chunks of files, then you should send them as binary, but you will need to chunk it yourself, and deal with potential ordering problems in the consumer (e.g. two clients are streaming the same filename, your key, at the same time, with interwoven values)

The recommendation for dealing with files (i.e. large binary content) is to not send them through Kafka, but rather upload them to a shared filesystem, then send the URI as a string through an event.

Upvotes: 2

Related Questions