CODEWITHSUNDEEP
distributed-computingapache-zookeeper
Sandeep Das
Sandeep Das

Reputation: 1050

Copy file using zookeeper

I have a distributed application and I use zookeeper to manage configuration data in all distributed servers.My service in each server needs some dlls to run . I am trying to build a centralized system from where I can copy my dlls to all the server. Can I achieve that using zookeeper ? I am aware that "ZooKeeper is generally not designed for large size storage" . My dll files are of size less the 3mb.

Upvotes: 2

Views: 2272

Answers (3)

15412s
15412s

Reputation: 3768

Zookeeper is designed to transfer messages inside the cluster. Best thing you can do is create a Znode_A that will contain Znodes, watch znode a for changes. Each Znode in Znode_A will represent a dll and will contain a dll path. Each node on the cluster watch for Znode_A data changes, so when a new dll (znode) will be created the nodes will know to copy the dll from a main repository.

Upvotes: 3

Aman Sehgal
Aman Sehgal

Reputation: 11

In order to transfer files you can use SCP. As data you can pass file path of your dlls. Using SCP you can pull files from base repository.

Upvotes: -1

kjw0188
kjw0188

Reputation: 3685

There is a 1mb soft limit on how large node data can get. According to the docs you can increase the max data size:

jute.maxbuffer:

(Java system property: jute.maxbuffer)

This option can only be set as a Java system property. There is no zookeeper prefix on it. It specifies the maximum size of the data that can be stored in a znode. The default is 0xfffff, or just under 1M. If this option is changed, the system property must be set on all servers and clients otherwise problems will arise. This is really a sanity check. ZooKeeper is designed to store data on the order of kilobytes in size.

I would not recommend using Zookeeper for this purpose, (you could much more easily host the binaries on a web server instead,) but it does seem possible in theory.

Upvotes: 4

Related Questions