CODEWITHSUNDEEP
linuxnetwork-programmingapache-spark
Dawid Pura
Dawid Pura

Reputation: 1029

Setting an IP address as spark.driver.host instead of hostname in Spark?

I have a problem with declaring an spark.driver.host setting. Assume that i have an host with a HOSTNAME and binded address 192.168.1.1 (its linux, windows has not this problem at all).

If i set a property spark.driver.hostname = 192.168.1.1 workers will respond to HOSTNAME, which is really bad, because they don't have DNS with this hostname.

As this troubleshooting page says:

If SPARK_LOCAL_IP is set to an IP address, it will be resolved to a hostname.

And here is a question: how to NOT setting this IP address as a hostname?

Upvotes: 2

Views: 7222

Answers (2)

tangtang
tangtang

Reputation: 71

you can set SPARK_LOCAL_HOSTNAME=ip address in your spark environment to solve the problem.

Upvotes: 7

Rafal Glowacki
Rafal Glowacki

Reputation: 11

hostname -f gives your FQDN add this to /etc/hosts file as

[your ip address] [your hostname] on spark nodes

or add your hostname to dns used by your spark nodes

Upvotes: 1

Related Questions