CODEWITHSUNDEEP
javamavenapache-sparklivy
snark
snark

Reputation: 2887

How to build the Livy client on Windows?

I want to write a Java app on Windows that uses Livy's Java API to talk to an Apache Livy server that's hosted remotely. The docs say:

Add the Livy client dependency to your application's POM:

<dependency>
  <groupId>org.apache.livy</groupId>
  <artifactId>livy-client-http</artifactId>
  <version>0.4.0-SNAPSHOT</version>
</dependency>

Note: Until Livy's first Apache release you will have to install the livy artifacts locally using mvn install

Unfortunately it looks like building Livy on Windows isn't supported. Has anyone successfully built the Livy client on Windows?

Upvotes: 1

Views: 2053

Answers (1)

snark
snark

Reputation: 2887

All I cared about was getting livy-client-http to build and installed in my local maven .m2 repository. This is what I did in Windows 7:

git clone incubator-livy then comment out this section in its pom.xml (credit to Yatzhash):

<requireOS>
    <family>unix</family>
</requireOS>

Run mvn install -DskipTests in the top level directory. If you do this from a regular Windows command prompt it will eventually fail with this error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (default) on project livy-server: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "bash" (in directory "C:\github.com\incubator-livy\server"): CreateProcess error=2, The system cannot find the file specified [ERROR] around Ant part ...... @ 4:27 in C:\github.com\incubator-livy\server\target\antrun\build-main.xml

But if you run the same command in a Git Bash prompt instead you can bypass this error. Eventually the build will fail at livy-integration-test but at least the livy-client-http build should have passed.

But you'll find this jar will have been installed in your .m2 repository: livy-client-http-0.4.0-incubating-SNAPSHOT.jar. This means you need to modify your own client app's dependency to look like this instead of the one recommended by the Livy docs:

<dependency>
    <groupId>org.apache.livy</groupId>
    <artifactId>livy-client-http</artifactId>
    <version>0.4.0-incubating-SNAPSHOT</version>
</dependency>

I also had to add this dependency to my client app because I wanted to talk to a Spark 2.1 server via Livy:

<dependency>
    <!-- See https://spark.apache.org/docs/2.1.0/programming-guide.html#linking-with-spark -->
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.1.0</version>
</dependency>

Then my client app would compile in Windows.

Upvotes: 1

Related Questions