liltitus27
liltitus27

Reputation: 1760

Can I use Spark's REST API to get the version of Spark on the Workers

I know I can get the version of Spark v2.2.1 that's running on Spark Master with this command:

http://<spark-master>:4040/api/v1/version

which will return something like

{
  "spark" : "2.2.1"
}

However, I also want to check the version of Spark running on each Worker. I know I can get a list of all Workers thusly:

http://<spark-master>:8080/json/

which will return a response similar to

{
    "url": "spark://<spark-master>:7077",
    "workers": [{
            "id": "worker-20180228071440-<ip-address>-7078",
            "host": "<ip-address>",
            "port": 7078,
            "webuiaddress": "http://<ip-address>:8081",
            "cores": 8,
            "coresused": 8,
            "coresfree": 0,
            "memory": 40960,
            "memoryused": 35875,
            "memoryfree": 5085,
            "state": "ALIVE",
            "lastheartbeat": 1519932580686
        }, ...
    ],
    "cores": 32,
    "coresused": 32,
    "memory": 163840,
    "memoryused": 143500,
    "activeapps": [{
            "starttime": 1519830260440,
            "id": "app-20180228070420-0000",
            "name": "<spark-app-name>",
            "user": "<spark-app-user>",
            "memoryperslave": 35875,
            "submitdate": "Wed Feb 28 07:04:20 PST 2018",
            "state": "RUNNING",
            "duration": 102328434
        }
    ],
    "completedapps": [],
    "activedrivers": [],
    "status": "ALIVE"
}

I'd like to use that information to query each Spark Worker's version. Is this possible?

Upvotes: 0

Views: 702

Answers (0)

Related Questions