CODEWITHSUNDEEP
javascalaapache-sparkapache-spark-mllib
user1371662
user1371662

Reputation: 103

Can Spark code in Scala be converted to Java?

I have source that uses Apache Spark. The source is written in Scala and I want to convert it to Java.

Is it possible to convert to Java from all Scala source?

Upvotes: 0

Views: 2065

Answers (4)

suztomo
suztomo

Reputation: 5202

No, there's no such tool that converts Scala source to Java.

Upvotes: 0

Soumya Simanta
Soumya Simanta

Reputation: 11741

It depends. Currently Spark (till version 1.1) provides a Java version for most of it's API/sub-projects, but not all. GraphX Java API is not supported. So if your Scala code is using graphx libraries then it may not work in Java unless you write custom glue code.

http://spark.apache.org/docs/latest/api/java/index.html

Even if you manage to decompile your Scala class files into Java you may not able to execute the Java code if your program semantics are not wrong. I would recommend you learn the Java API and then rewrite your Scala code in Java. This will also give you a better understanding of the Spark API, which is essential if you want to write your own Spark programs in the future.

Upvotes: 1

maasg
maasg

Reputation: 37435

No. The Spark Scala API is different from the Java API.

tl;dr;

Although one could do tricks to compile/decompile or have some IDE do the translation, the resulting code will not work in Spark.

One of the reasons is that the Spark Scala API makes use of implicit conversions to enable type-dependent functionality that is not available in the Java API, where the use of explicit classes is required.

A simple example:

Scala

val pairs = words.map(w => (w,1))  // count here is of type RDD[(String,Int)]
val counts = words.reduceByKey(_ + _)

Java

JavaPairRDD<String, Integer> pairs = words.mapToPair(new PairFunction<String, String, Integer>() {
  public Tuple2<String, Integer> call(String s) { return new Tuple2<String, Integer>(s, 1); }
});
JavaPairRDD<String, Integer> counts = pairs.reduceByKey(new Function2<Integer, Integer>() {
  public Integer call(Integer a, Integer b) { return a + b; }
});

Note how the Java counterpart needs to use specific methods (map in Scala vs mapToPair in Java) and specify a given class (JavaPairRDD) to gain access to the specific pair functions, while in Scala that's not required.

Upvotes: 3

dawrutowicz
dawrutowicz

Reputation: 3020

Compile sources to .class files (or download it if avaliable) and give a try to decompile it.

Here is one of such tools with wiki how-to:

https://bitbucket.org/mstrobel/procyon/wiki/Java%20Decompiler

Upvotes: 0

Related Questions