CODEWITHSUNDEEP
javaapache-drill
ztv
ztv

Reputation: 134

Apache Drill: Multiple Drillbits on same server

I am trying to setup a Drill cluster with multiple Drillbits locally on my computer.

I have Zookeeper running with three nodes (192.168.1.2:2181, 192.168.1.3:2181, 192.168.1.4:2181).

I have 3 separate config directories containing each different drill-env.sh and drill-override.conf.
Here are the main contents of dril-env.sh:

export DRILL_HOST_NAME=node1
export DRILL_IDENT_STRING="node1"
export DRILL_PID_DIR=${DRILL_PID_DIR:-"<path_to_pid_dir>/pid_dir1/"}

export DRILL_HOST_NAME=node2
export DRILL_IDENT_STRING="node2"
export DRILL_PID_DIR=${DRILL_PID_DIR:-"<path_to_pid_dir>/pid_dir2/"}

export DRILL_HOST_NAME=node3
export DRILL_IDENT_STRING="node3"
export DRILL_PID_DIR=${DRILL_PID_DIR:-"<path_to_pid_dir>/pid_dir3/"}

and drill-override.conf:

drill.exec: {
  cluster-id: "mycluster",
  zk.connect: "192.168.1.2:2181,192.168.1.3:2181,192.168.1.4:2181",
}

Also, I have added in etc/hosts:

192.168.1.2 node1
192.168.1.3 node2
192.168.1.4 node3

I start the Drillbits using

/bin/drillbit.sh --config <path_to_conf>/conf1/ start
/bin/drillbit.sh --config <path_to_conf>/conf2/ start
/bin/drillbit.sh --config <path_to_conf>/conf3/ start

However, only the first Drillbit starts normally, the other ones fail with following exception:

Exception in thread "main" org.apache.drill.exec.exception.DrillbitStartupException: Failure during initial startup of Drillbit.
    at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:590)
    at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:556)
    at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:552)
Caused by: org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Drillbit could not bind to port 31010.

Server type: UserServer

[Error Id: bb497859-bf3e-43b7-bad7-dfa295122068 ]
    at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:688)
    at org.apache.drill.exec.rpc.BasicServer.bind(BasicServer.java:211)
    at org.apache.drill.exec.service.ServiceEngine.start(ServiceEngine.java:100)
    at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:227)
    at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:586)
    ... 2 more
Caused by: java.net.BindException: Address already in use
    at java.base/sun.nio.ch.Net.bind0(Native Method)
    at java.base/sun.nio.ch.Net.bind(Net.java:555)

Anyone know how I can fix this?

Is it not pssible to bind a specific IP address to each Drillbit?

I know that Zookeeper is not a problem since I can use it for my Spark cluster without problems and zkServer.sh status shows it's running.
I also tried allow_loopback_address_binding: true, but same result.
Also, I know I could probably just use different ports for each Drillbit, but I'd prefer a solution using the same ports.

Update:
Apparently they are working on this issue and in the upcoming Drill version 1.21.1, it should be possible to bind the address to Drillbits (https://issues.apache.org/jira/browse/DRILL-8409)

Upvotes: 0

Views: 176

Answers (1)

Dzamo Norton
Dzamo Norton

Reputation: 1389

You can configure the ports that Drillbits bind to to avoid the port conflicts you encountered, but not yet the IP addresses. Another way to run a Drill cluster on a single machine is to use the Docker Compose scripts in the Drill source repo.

Upvotes: -1

Related Questions