CODEWITHSUNDEEP
apache-kafkakafka-topic
Giorgi Tsiklauri
Giorgi Tsiklauri

Reputation: 11120

Kafka Topic vs. Kafka Log - What is the difference between these two?

I often get to hear, read and see in a lot of tutorials, text guides or videos, that in the realm of Apache Kafka:

topics are just logs.

I understand, that when we're mentioning logs in Kafka, we're not talking about logging systems (like Slf4j or etc.), and I also understand, that Kafka log is a name for an ordered collection of events (and topic is also that, as far as I know).

In the KAFKA_HOME\config\server.properties file, I even see this:

# A comma separated list of directories under which to store log files
log.dirs=C:/tmp/kafka-logs

But I still struggle to clearly differentiate Kafka topics and Kafka logs and understand what is the difference between those two.

If there are two names, there should be some differences.. or it's absolutely interchangeably terms?

P.S. I have seen this and it doesn't really help.

Upvotes: 3

Views: 534

Answers (1)

Zazaeil
Zazaeil

Reputation: 4119

A "log" in programming - and it is not narrowed to Kafka only - is an immutable somehow ordered sequence of events (rather than a collection as you stated - which lacks ordering and generally open for mutations).

So, a "transaction log" is an ordered sequence of transactions, which, once commited, can't be removed. On practice, however, there must exist an internal mechanism how to eventually wipe out longly outdated data otherwise the memory overflow is a question of time. In Kafka the "log compaction" services this purpose.

While Kafka's topics are specific to it. These are just named pieces, which you can address within a Kafka cluster, like foo.bar is a perfectly fine name for a topic. It would be a logical name of the physical "partitions" of which this topic is built up and which actually store the (log) data at a disk.

Upvotes: 2

Related Questions