Kafka-stream Deployment strategy

Question

Team, What strategy should be followed for Kafka-streams app deployment?

We have a cluster of app servers and as per deployment strategies we used Jenkins CI/CD for jar deployment. Here come re-balancing issues. After deployment of app, within 3-5 minuts all the services get up and group gets in re-balancing state.

before deployment snapshot of the assigned partition and consumers.

TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
METRICS                        62         70306073        70307980        1907       METRICS-0d65d2ba-06ce-4488-bd4b-7509391d6f08-StreamThread-5-consumer-540b76ab-8374-4e99-8507-c3003488c61b/172.24.xx.xx                  METRICS-0d65d2ba-06ce-4488-bd4b-7509391d6f08-StreamThread-5-consumer
METRICS                        30         73200612        73202022        1410       METRICS-ef9ac162-5e7f-4695-88c4-0660485a3f29-StreamThread-2-consumer-20919608-c5e9-4263-852b-04ae7f16bcf9/172.24.xy.xy                  METRICS-ef9ac162-5e7f-4695-88c4-0660485a3f29-StreamThread-2-consumer
METRICS                        31         75197907        75199506        1599       METRICS-784f2d93-9cee-445f-ae19-1eea98a2dd11-StreamThread-3-consumer-6211abf7-47b6-443d-8047-301fbd6a0ebf/172.24.xz.xz                  METRICS-784f2d93-9cee-445f-ae19-1eea98a2dd11-StreamThread-3-consumer

and after deployment/restart the partition aligned to different consumers:

TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID

METRICS                        62         72423124        72424080        956        METRICS-7617b646-a9f8-4cee-ad40-3ace6e1cca1e-StreamThread-4-consumer-687ac6fb-5915-478a-87f6-25204f8261dc/172.24.3x.xx                  METRICS-7617b646-a9f8-4cee-ad40-3ace6e1cca1e-StreamThread-4-consumer

METRICS                        30         74813289        74814057        768        METRICS-2535d442-1f58-4c1a-bbd5-c0ef46a074d7-StreamThread-3-consumer-0496332f-4a60-42d3-98e9-d1ef9ef7e26a/172.24.3x.xx                  METRICS-2535d442-1f58-4c1a-bbd5-c0ef46a074d7-StreamThread-3-consumer

To Overcome with this situation we had tried the set kafka-consumer timeout strategy to avoid the same but it seems not worked for me.

/XXX/kafka-1.0.0/bin/kafka-consumer-groups.sh --bootstrap-server 172.29.XX.XXX:9092 --describe --group GROUPNAME --timeout 300000

TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
GROUPNAME                        69         73592702        73593636        934        METRICS-0ae9cabe-4879-4400-b111-84580ea3118b-StreamThread-2-consumer-021397b6-87b3-49fb-ae01-b0acbc54cfc5/172.24.xx.xx                  METRICS-0ae9cabe-4879-4400-b111-84580ea3118b-StreamThread-2-consumer

So, it will be good if any one help us to get the sticky assignment(partition should be assigned to same node as before deployment) of partitions after deployment changes. Thanks in advance!

Kafka-stream Deployment strategy

Answers (1)

Related Questions