CODEWITHSUNDEEP
dockerdocker-composeapache-nifiapache-zookeeper
Sascha
Sascha

Reputation: 11

Apache NiFi Cluster in Docker over 3 VM's

I want to make a NiFi cluster in docker over 3 vm's. I found a docker-compose file that creates a cluster on one node and try to edit this file.

I found out that i need zookeeper, but do i need one zookeeper each instance? and what ports should i open or map in docker?

the docker-compose file i found:

version: "3"
services:
  zookeeper:
    hostname: zookeeper
    container_name: zookeeper
    image: zookeeper:3.6.1
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  nifi:
    image: apache/nifi:1.11.4
    ports:
      - 8080 # Unsecured HTTP Web Port
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zookeeper:2181
      - NIFI_ELECTION_MAX_WAIT=1 min

I changed the file like this (on each VM the ip is correct )

    version: "3"
    services:
  zookeeper:
    hostname: zookeeper
    container_name: zookeeper
    image: 'zookeeper:3.6.1'
    ports:
      - 2181
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  nifi:
    image: apache/nifi:1.11.4
    ports:
      - 8080 # Unsecured HTTP Web Port
      - 8082
      - 9001
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
#      - NIFI_ZK_CONNECT_STRING=zookeeper:2181
      - NIFI_ZK_CONNECT_STRING=192.168.2.10:2181,192.168.2.20:2181,192.168.2.30:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
      - NIFI_CLUSTER_ADDRESS=192.168.2.XX

and in the logs i found this message but cant find any solution

ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Upvotes: 1

Views: 1090

Answers (1)

Bronislav
Bronislav

Reputation: 11

I found out that i need zookeeper, but do i need one zookeeper each instance?

No, you can use one zookeeper

and what ports should i open or map in docker?

as I know you need 2888, 3888, 2181 to be open. But only 2181 to communicate with nifi 2888 and 3888 for zookeeper cluster communication.

Upvotes: 1

Related Questions