CODEWITHSUNDEEP
apache-sparkpyspark
Stuart Berg
Stuart Berg

Reputation: 18151

Spark: Determine the driver address from within a task

For debugging purposes, I want to send a message from a spark task (running within an executor) to the driver machine. The docs claim that spark.driver.host contains this information, but I don't know how to access it from worker (or from the driver, for that matter...)

Upvotes: 2

Views: 2022

Answers (2)

Stuart Berg
Stuart Berg

Reputation: 18151

Just pass the driver's IP to the workers via a captured global variable:

import socket
driver_ip_addr = socket.gethostbyname(socket.gethostname())

def process_data(data):
    # Do some work ...
    send_log_msg(driver_ip_addr, "hello!")
    # Do some more work...

processed_data = my_rdd.map(process_data).collect()

Upvotes: 2

Viacheslav Rodionov
Viacheslav Rodionov

Reputation: 2345

sc.getConf.get("spark.driver.host")

but driver is not what you need. You need a worker. So try something like this code inside your rdd.mapPartitions block:

val localhost = java.net.InetAddress.getLocalHost
val ip = java.net.NetworkInterface.getByInetAddress(localhost)

check this for a more precise approach.

Upvotes: 3

Related Questions