CODEWITHSUNDEEP
node.jsdockerdocker-composeapache-pulsar
RATTLESNAKE
RATTLESNAKE

Reputation: 3228

Apache-Pulsar cluster setup (multi broker) in local machine

I am able to run apache-pulsar standalone in local machine as docker container.

docker-compose.yml

pulsar-standalone:
    image: apachepulsar/pulsar
    ports:
      - "8080:8080"
      - "6650:6650"
    expose:
      - 8080
      - 6650
    environment:
      - PULSAR_MEM=" -Xms512m -Xmx512m -XX:MaxDirectMemorySize=1g"
    command: >
      /bin/bash -c
      "bin/apply-config-from-env.py conf/standalone.conf
      && bin/pulsar standalone"

Can anyone let me know how I can run multi broker setup for the same?

Thanks!

Upvotes: 1

Views: 2040

Answers (3)

visortelle
visortelle

Reputation: 1029

You can use the Puls CLI tool: https://github.com/tealtools/puls It allows you to easily spin up local Pulsar instances with multiple clusters and multiple brokers per cluster.

puls create --num-clusters 2 --num-brokers 3 --num-bookies 2 my-instance
puls start my-instance

The command above will start a new Pulsar instance with two clusters, three brokers per cluster, and three bookies per cluster.

Upvotes: 0

ConfusedSoul
ConfusedSoul

Reputation: 87

Based on @Sergii's answer, I created a docker-compose file above in a manner similar to the kafka stack depoyment.

The necessary configuration and yaml can be found in this repo

This is a single pulsar cluster setup containing:

  1. 3 node ZooKeeper cluster
  2. 3 node BookKeeper cluster
  3. 2 Pulsar broker nodes
  4. 1 Pulsar proxy to allow access to pulsar from outside the docker-compose setup.

The setup uses apachepulsar/pulsar:2.4.1 docker image.

Upvotes: 2

Sergii Zhevzhyk
Sergii Zhevzhyk

Reputation: 4202

My assumption is that by "multi-broker setup" you mean Pulsar cluster. It is needed a bit more than a couple of brokers to achieve this goal. To create a Pulsar cluster the following tasks should be completed:

  • Deploy ZooKeeper cluster

  • Initialize Cluster metadata

  • Deploy BookKeeper cluster

  • (Finally) Deploy one or more Pulsar brokers

I am not aware of any existing docker-compose configurations to deploy a Pulsar cluster, but there a plenty of options to deploy in Kubernetes. For testing on a local machine, it is possible to start a Pulsar cluster in minikube using helm.

Upvotes: 0

Related Questions