CODEWITHSUNDEEP
apache-kafkaconfluent-platformksqldb
zimvak
zimvak

Reputation: 331

How to select/assign partition in KSQL

I believe this is a related question: Using Kafka KSQL to select all events of a topic from a specific partition with given offset

How do you select/assign a partition via KSQL? I'm trying to prevent KSQL from reading from all partitions, as the necessary data only lives in one shard.

For example:

CLI v5.4.1, Server v5.4.1

SET 'auto.offset.reset'='earliest';
CREATE STREAM SOURCE_STREAM (FIELD_1 BIGINT)
    WITH (
        VALUE_FORMAT='AVRO',
        KAFKA_TOPIC='source_topic',
        PARTITIONS=2,
        REPLICAS=1
    );

Insert some mock data which lives in partition 0 and partition 1 (not really assigned but for example)

INSERT INTO SOURCE_STREAM (FIELD_1) VALUES (123);  # say in partition 0
INSERT INTO SOURCE_STREAM (FIELD_1) VALUES (456);  # say in partition 1

With a consumer API one could do the following:

consumer.assign(TopicPartition(topic=source_topic, partition=0))
consumer.assign(TopicPartition(topic=source_topic, partition=1))
consumer.get()

However, with the current API I'm unsure how to "assign" a partition either at the client level or at the server property level. The below derivative stream would read from all partitions:

CREATE STREAM DERIVATIVE_STREAM AS 
    SELECT
        FIELD_1
    FROM SOURCE_STREAM
    EMIT CHANGES;

EXPLAIN CSAS_DERIVATIVE_STREAM_n;

(I know I can use a WHERE statement to filter the data but I want to explicitly read from partition 0|1)

Upvotes: 0

Views: 1015

Answers (1)

Robin Moffatt
Robin Moffatt

Reputation: 32080

ksqlDB doesn't work this way. You use SQL to declare what you want, not how you want it.

As you said in your question, you can use WHERE to apply a predicate to your query, and can use ROWKEY to target the message key value.

I guess the parallel in the RDBMS world would be an execution plan hint for the cost-based optimiser.

If you want to log this as an enhancement request to ksqlDB please do so here: https://github.com/confluentinc/ksql/issues/new

Upvotes: 1

Related Questions