CODEWITHSUNDEEP
clickhouse
dsr301
dsr301

Reputation: 779

Understanding clickhouse partitions

I see that clickhouse created multiple directories for each partition key(in each node). Documentation says the directory name format is : partition ID_minimum block number_maximum block number_level. Any idea what is level here? 347 distinct partition keys on one node(for one table) created 1358 directories. (custom partitioning)

The documentation recommends not to have more than 1000 partitions. Should we just keep in mind the number of partitions keys or the number of directories also?

Also, Is there a configuration on how to control this number of directories?

enter image description here

Upvotes: 7

Views: 18988

Answers (2)

Divyanshu Jimmy
Divyanshu Jimmy

Reputation: 2752

Adding to this discussion, you can check parts and partition in the following ways:

For active partitions:

select count(distinct partition) from system.parts where table in ('table_name') and active

For active parts:

select count() from system.parts where table in ('table_name') and active

Inactive parts will be removed soon in less than 10 minutes.

To view table parts and partition together:

SELECT
    partition,
    name,
    active
FROM system.parts
WHERE table = 'table_name'

You can also read more about parts, partition and how merging happens here.

Upvotes: 3

Amos
Amos

Reputation: 3276

Any idea what is level here?

Level is a concept of LSM-tree. MergeTree tables have mechanisms to merge data parts into bigger and deeper (w.r.t level) ones.

Should we just keep in mind the number of partitions keys or the number of directories also?

Well I don't think that's a good idea as this method doesn't scale well. You'd better choose a low-cardinality column or expression as the partition key.

Also, Is there a configuration on how to control this number of directories?

No explicit settings for that. But you can easily use modular expression to limit the total number of partitions.

Upvotes: 2

Related Questions