CODEWITHSUNDEEP
apache-sparkclouderalivy
Lokesh Yadav
Lokesh Yadav

Reputation: 998

How to set proxy user in Livy Job submit through its Java API

I am using Livy's Java API to submit a spark job on YARN on my cluster. Currently the jobs are being submitted as 'livy' user, but I want to submit the job as a proxy user from Livy.

It is possible to do this by sending POST request to the Livy server, by passing a field in the POST data. I was thinking if this could be done by Livy's Java API.

I am using the standard way to submit a Job:

LivyClient client = new LivyClientBuilder()
  .setURI(new URI(livyUrl))
  .build();

try {
  System.err.printf("Uploading %s to the Spark context...\n", piJar);
  client.uploadJar(new File(piJar)).get();

  System.err.printf("Running PiJob with %d samples...\n", samples);
  double pi = client.submit(new PiJob(samples)).get();

  System.out.println("Pi is roughly: " + pi);
} finally {
  client.stop(true);
}

Upvotes: 1

Views: 2273

Answers (1)

Lokesh Yadav
Lokesh Yadav

Reputation: 998

Posting answer to my own question. Currently there is no way to set the proxy user through the LivyClientBuilder.

A workaround for this is:

  1. Create the session through the REST API (POST request to < livy-server >/session/ ) and read the session ID from the request's response. Proxy user can be set via the REST API by passing it in the POST data: {"kind": "spark", "proxyUser": "lok"}
  2. Once the session is created, connect to it using the ID via LivyClientBuilder ( livyURL would be < livy-server >/sessions/< id >/ ).

Upvotes: 2

Related Questions