CODEWITHSUNDEEP
javaapache-sparklivyapache-knox
Makrushin Evgenii
Makrushin Evgenii

Reputation: 1181

How to submit a Job to Livy (under Knox) with Programmatic API?

This code from Livy Docs worked fine:

LivyClient client = new LivyClientBuilder()
    .setURI(new URI("http://sample.ru:8999"))
    .build();
String piJar = "C:\\Users\\e.makrushin\\IdeaProjects\\untitled-maven\\out\\artifacts\\untitled_maven_jar\\untitled-maven.jar";
try {
    String date = "2020.01.07";
    System.err.printf("Job jar file %s uploading to the Spark context...\n", piJar);
    client.uploadJar(new File(piJar)).get();
    System.err.println("Job running at date " + date + " ...");
    double cheques = client.submit(new ChequesJob(date)).get();
    System.out.println("Job finished. " + cheques + " cheques found.");
} finally {
    client.stop(true);
}

DevOps routed a Livy gateway through Knox:

<service role="LIVYSERVER" name="livy" version="0.1.0">
  <routes>
    <route path="/livy/v1/sessions">
        <rewrite apply="LIVYSERVER/livy/addusername/inbound" to="request.body"/>
    </route>
    <route path="/livy/v1/**?**"/>
    <route path="/livy/v1"/>
    <route path="/livy/v1/"/>
  </routes>
  <dispatch classname="org.apache.knox.gateway.livy.LivyDispatch"/>
</service>

rewrite.xml:
<rules>
  <rule name="LIVYSERVER/livy/user-name">
    <rewrite template="{$username}"/>
  </rule>
  <rule dir="IN" name="LIVYSERVER/livy/root/inbound" pattern="*://*:*/**/livy/v1">
    <rewrite template="{$serviceUrl[LIVYSERVER]}"/>
  </rule>
  <rule dir="IN" name="LIVYSERVER/livy/path/inbound" pattern="*://*:*/**/livy/v1/{path=**}?{**}">
    <rewrite template="{$serviceUrl[LIVYSERVER]}/{path=**}?{**}"/>
  </rule>
  <filter name="LIVYSERVER/livy/addusername/inbound">
    <content type="*/json">
      <apply path="$.proxyUser" rule="LIVYSERVER/livy/user-name"/>
    </content>
  </filter>
</rules>rewrite template="{$serviceUrl[LIVYSERVER]}/"/>
    </rule>
    <rule dir="IN" name="LIVYSERVER/livy/inbound/path" pattern="*://*:*/**/livy/{**}">
        <rewrite template="{$serviceUrl[LIVYSERVER]}/{**}"/>
  </rule>
</rules>

Livy session list is now available at https://knox-gateway:8443/gateway/default/livy/v1/session

After that, as expected, this code stopped working:

Exception in thread "main" java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at org.apache.livy.client.http.HttpClient.propagate(HttpClient.java:182)
    at org.apache.livy.client.http.HttpClient.<init>(HttpClient.java:82)
    at org.apache.livy.client.http.HttpClientFactory.createClient(HttpClientFactory.java:37)
    at org.apache.livy.LivyClientBuilder.build(LivyClientBuilder.java:130)
    at Main.main(Main.java:14)

I cannot find anything in the Livy documentation about setting up authorization using the Programmatic API. What should I add to the call code or configuration files to make this code work again?

Upvotes: 0

Views: 764

Answers (1)

lmccay
lmccay

Reputation: 396

This appears to be a TLS/SSL trust store issue. I believe Livy is attempting to make a call to the server and is being presented an SSL cert that it does not trust. You should look for TLS configuration documentation rather than setting up authorization. Perhaps the following will help. https://community.cloudera.com/t5/Community-Articles/Enable-SSL-for-Livy-Server/tac-p/246070

Upvotes: 1

Related Questions