CODEWITHSUNDEEP
javascalaapache-spark
Olivier GRACIANNE
Olivier GRACIANNE

Reputation: 1

Is there a way to force waiting for the end of a java process to end in Scala or Spark?

In a Scala app, deployed through Spark, I have a code line which calls to a Java function executing native C++ code through a JNI. This call takes time, and if it is not the only one of his kind running, a resource usage conflict appears with a *** stack smashing detected ***: <unknown> terminated.

Here's the call, and it's scope :

[spark RDD].mapPartitionsWithIndex(f = (index: Int, it: Iterator[Row]) => {
  val sourceData: String = it.mkString()

  val result: List[List[String]] = new WrapperClass(sourceData, [misc parameters).getResult

  [wrinting result to a file]
}).take(1)

My WrapperClass.getResult, very simple, looks like this :

[java call related variables initialization]

UnitexJni.execUnitexTool("UnitexTool {InstallLingResourcePackage -p " + appliedPkg + " -x " + resDir + "} " + "{" + runScriptCmd + "} " + "{InstallLingResourcePackage -p " + appliedPkg + " -x " + resDir + " --uninstall}")

[retrieving, formatting and returning result]

The UnitexJni.execUnitexTool() is the java call.

So I would like to know if there is a way to force to wait until the end of this process before calling it over agin using Scala, Java or Spark functionality.

Upvotes: 0

Views: 447

Answers (1)

Moustafa Mahmoud
Moustafa Mahmoud

Reputation: 1590

You could use sys.process._ You will pass the shell script to the below process function with the script path. Also, you need to handle the shell script to have a return code. For example, If 0 success else failed. Please take care of the ! at the end of the line. You could also check more details to run quick command lines from this tutorial 

import scala.sys.process.Process
val externalShellScript = Process("sh", Seq(scriptPath)).!    
  if (externalShellScript != 0) {
    throw new Exception(
      "Error in executing external shell script from " + scriptPath)
  }

The Spark job will not continue unless this process finish. Below is simple shell script and the output.

touch test.txt
echo "any massage"

output in the console will be 

any massage

Upvotes: 1

Related Questions