CODEWITHSUNDEEP
apache-kafka-streamsrocksdb
adpap
adpap

Reputation: 209

What are the folders generated by RocksDB

In my kubernetes container I have configured my rocksdb for my kafka streams application

containers:
  ...
  volumeMounts:
            - mountPath: /tmp/state
              name: state-volume
volumes:
  - name: state-volume
    emptyDir:
    medium: Memory

After the deployment if I go to the location cd /tmp/state/my-service I see a dozen of folders. E.g. 0_105, 0_107, 0_9 etc. If I go to one of these folders e.g. cd /tmp/state/my-service/0_105/rocksdb/state I see the actual .sst files. Inside each of these 0_* folders there are multiple sst files which is how rocksdb splits the files keys I suppose. But what do those 0_* folders do and how are they splitted the way they are?

Upvotes: 1

Views: 1226

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62330

Kafka Streams create "tasks" based on your program structure and the number of input topic partitions. The directories you sees are task directories that allow each task to store its state locally in an isolated manner.

Each task has a task id Y_Z that is used as task directory name. The first number is the sub-topolgoy ID and the second number is basically the input partition number.

You can see what sub-topologies are generated by describing your topology via Topology#describe() (you can just call print the `TopologyDescription' to stdout if you want).

Upvotes: 2

Related Questions