CODEWITHSUNDEEP
apache-zookeepermesos
doktoroblivion
doktoroblivion

Reputation: 428

New 3.5 Zookeeper nodes do not appear to be dynamically added to my cluster

I am using Zookeeper version: 3.5.3-beta and had it configured in a standard 3 node configuration. I am now trying to add 3 new nodes to it following the documentation, however the new nodes do not appear to be getting dynamically added as observers (see https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html#ch_reconfig_dyn).

First, I tried to add the first node using the following setting:"ZOOKEEPER_IP_LIST=10.144.81.81 10.144.81.83 10.144.81.86 10.144.46.54" , but it does not join the cluster. When I start up the new node I still see the following, only showing the 3 initial nodes in the cluster. I even tried restarting the 3 initial nodes to include the new nodes, but it still reflected the old configuration.

clientPort=2181
secureClientPort=-1
dataDir=/dbpath/data/version-2
dataDirSize=134254305
dataLogDir=/dbpath/data/version-2
dataLogSize=134254305
tickTime=2000
maxClientCnxns=30
minSessionTimeout=4000
maxSessionTimeout=40000
serverId=3
initLimit=60
syncLimit=5
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
membership: 
server.0=10.144.81.81:2888:3888:participant
server.1=10.144.81.83:2888:3888:participant
server.2=10.144.81.86:2888:3888:participant

When that did not work I decided to try the zkCli.sh command on the leader first to see if the configuration tried above matched the results from the REST interface, it did.

$ sudo bin/zkCli.sh -server 10.144.81.83:2181 config
...
2017-11-09 12:13:38,172 [myid:10.144.81.83:2181] - INFO  [main-SendThread(10.144.81.83:2181):ClientCnxn$SendThread@1381] - Session establishment complete on server 10.144.81.83/10.144.81.83:2181, sessionid = 0x100301c01db000a, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
server.0=10.144.81.81:2888:3888:participant
server.1=10.144.81.83:2888:3888:participant
server.2=10.144.81.86:2888:3888:participant
version=100000000

I then attempted to use zkCli.sh to reconfigure and add the one new node, but it throws an Authentication issue, which is confusing because I have not configured ACLs or any authentication that I know of that would cause this error.

$ sudo bin/zkCli.sh -server 10.144.81.83:2181 reconfig -file ../newconfig.cfg.dynamic
/usr/bin/java
Connecting to 10.144.81.83:2181
2017-11-09 12:03:32,729 [myid:] - INFO  [main:Environment@109] - Client environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60, built on 04/03/2017 16:19 GMT
2017-11-09 12:03:32,735 [myid:] - INFO  [main:Environment@109] - Client environment:host.name=DockerNode-master-cloudeng-20170925-1652-Jy1.localdomain
2017-11-09 12:03:32,739 [myid:] - INFO  [main:Environment@109] - Client environment:java.version=1.7.0
2017-11-09 12:03:32,739 [myid:] - INFO  [main:Environment@109] - Client environment:java.vendor=IBM Corporation
2017-11-09 12:03:32,739 [myid:] - INFO  [main:Environment@109] - Client environment:java.home=/opt/ibm/ibm-java-x86_64-70/jre
2017-11-09 12:03:32,739 [myid:] - INFO  [main:Environment@109] - Client environment:java.class.path=/home/ibmadmin/zookeeper-3.5.3-beta/bin/../build/classes:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../build/lib/*.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Final.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/log4j-1.2.17.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jline-2.11.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-util-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-server-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-security-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-io-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jetty-http-9.2.18.v20160721.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/javax.servlet-api-3.1.0.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jackson-mapper-asl-1.9.11.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/jackson-core-asl-1.9.11.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../lib/commons-cli-1.2.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../zookeeper-3.5.3-beta.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../src/java/lib/*.jar:/home/ibmadmin/zookeeper-3.5.3-beta/bin/../conf:
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:java.library.path=/opt/ibm/ibm-java-x86_64-70/jre/lib/amd64/compressedrefs:/opt/ibm/ibm-java-x86_64-70/jre/lib/amd64:/usr/lib64:/usr/lib
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:java.compiler=j9jit26
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:os.name=Linux
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:os.arch=amd64
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:os.version=4.4.0-89-generic
2017-11-09 12:03:32,740 [myid:] - INFO  [main:Environment@109] - Client environment:user.name=root
2017-11-09 12:03:32,741 [myid:] - INFO  [main:Environment@109] - Client environment:user.home=/root
2017-11-09 12:03:32,741 [myid:] - INFO  [main:Environment@109] - Client environment:user.dir=/home/ibmadmin/zookeeper-3.5.3-beta
2017-11-09 12:03:32,743 [myid:] - INFO  [main:Environment@109] - Client environment:os.memory.free=1MB
2017-11-09 12:03:32,744 [myid:] - INFO  [main:Environment@109] - Client environment:os.memory.max=256MB
2017-11-09 12:03:32,744 [myid:] - INFO  [main:Environment@109] - Client environment:os.memory.total=4MB
2017-11-09 12:03:32,749 [myid:] - INFO  [main:ZooKeeper@865] - Initiating client connection, connectString=10.144.81.83:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@f9b15d67
2017-11-09 12:03:32,765 [myid:] - INFO  [main:ClientCnxnSocket@236] - jute.maxbuffer value is 4194304 Bytes
2017-11-09 12:03:32,785 [myid:10.144.81.83:2181] - INFO  [main-SendThread(10.144.81.83:2181):ClientCnxn$SendThread@1113] - Opening socket connection to server 10.144.81.83/10.144.81.83:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-09 12:03:32,802 [myid:10.144.81.83:2181] - INFO  [main-SendThread(10.144.81.83:2181):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /10.144.81.83:59114, server: 10.144.81.83/10.144.81.83:2181
2017-11-09 12:03:32,812 [myid:10.144.81.83:2181] - INFO  [main-SendThread(10.144.81.83:2181):ClientCnxn$SendThread@1381] - Session establishment complete on server 10.144.81.83/10.144.81.83:2181, sessionid = 0x100301c01db0008, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
Authentication is not valid : 

$ cat ../newconfig.cfg.dynamic
server.0=10.144.81.81:2888:3888:participant
server.1=10.144.81.83:2888:3888:participant
server.2=10.144.81.86:2888:3888:participant
server.3=10.144.46.54:2888:3888:observer

I was under the impression by the documentation, either rightly or wrongly, that I could just specify the new topology changes without any specific node configuration changes. I would prefer not to use zkCli since it would just complicate my deployment process but if I have to I will.

Upvotes: 0

Views: 969

Answers (1)

Rico
Rico

Reputation: 61561

What's happening is that the new observer is not allowed the path where zookeeper keeps info about the config. Initially all the nodes add themselves in the static config, but any dynamic new node is not allowed because it doesn't have permissions to the path (/config)

You can try restarting your nodes with this jvm option:

-Dzookeeper.skipACL=true

Or you can become superuser:

 $ export ZK_CLASSPATH=/etc/zookeeper/conf/:/usr/hdp/current/zookeeper-server/lib/*:/usr/hdp/current/zookeeper-server/*
 $ java -cp $ZK_CLASSPATH org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:super123

Output:

super:super123->super:UdxDQl4f9v5oITwcAsO9bmWgHSI=

Then as root:

$ export SERVER_JVMFLAGS=-Dzookeeper.DigestAuthenticationProvider.superDigest=super:UdxDQl4f9v5oITwcAsO9bmWgHSI=
$ bin/zkCli.sh -server 10.144.81.83:2181 reconfig -file ../newconfig.cfg.dynamic

Hope it helps.

More info here: https://community.hortonworks.com/articles/29900/zookeeper-using-superdigest-to-gain-full-access-to.html

Upvotes: 1

Related Questions