CODEWITHSUNDEEP
javahadoopapache-spark
J.Done
J.Done

Reputation: 3033

Spark submit in java(SparkLauncher)

I made spark+hadoop yarn enviroment and spark-submit command works well. So I made SparkLauncher java code to do this in my application jar, BUT somehow it doesn't work (actually computer fan is spinning at first but not as long as i did with spark-submit.)

It seems not work well (no application log in hadoop web ui, unlike spark-submit). I cannot see any error log when I do with 'SparkLauncher'. without log message, I can do nothing with it.

Here is how I made it so far.

public class Main {
    public static void main(String[] args) {
        Process spark = null;
        try
        {
            spark = new SparkLauncher()
            .setAppResource("/usr/local/spark/examples/jars/spark-examples*.jar")
            .setMainClass("org.apache.spark.examples.SparkPi")
            .setMaster("yarn")
            .setDeployMode( "cluster")
            .launch();
        }
        catch( IOException e)
        {
            e.printStackTrace();
        }
    }
}

executed it with ( java -jar example.jar)

Upvotes: 2

Views: 1364

Answers (1)

SCouto
SCouto

Reputation: 7928

I had the same problem at first. I think the main issue is that you forgot about the waitFor().

Also, it's really helpfull to extract your errorMessage and deal with it (e.g. log it or checking it while debuging ) within your java code. To allow this, you should create a streamReader thread as follows:

    InputStreamReaderRunnable errorStreamReaderRunnable = new InputStreamReaderRunnable(spark.getErrorStream(), "error");
    Thread errorThread = new Thread(errorStreamReaderRunnable, "LogStreamReader error");
    errorThread.start();

   int result= spark.waitFor();
   if(result!=0) {
            String errorMessage = extractExceptionMessage(errorStreamReaderRunnable.getMessage());
            LOGGER.error(errorMessage);
        }

This should be after your launch() command and inside your try block. Hope it helps

Upvotes: 3

Related Questions