Aavash Bhandari
Reputation: 117
Spark-submit on GCP dataproc shows java.io. FileNotFoundException error
All the cases are performed in GCP data proc cluster with a master and 4 executors.
Case 1: When I execute a runnable JAR file with the spark-submit in Local mode with spark-config as
SparkConf config = new SparkConf().setMaster("local[*]").setAppName("ANNCLUSTERED");
spark-submit /home/aavashbhandari/dataset/RunSCL.jar /home/aavashbhandari/dataset/California_Nodes.txt /home/aavashbhandari/dataset/California_Edges.txt /home/aavashbhandari/dataset/California_part_4.txt 4 10000 10000 1
and run it on local mode, it runs without any issues.
Case 2: Setting the spark configuration on the source as:
SparkConf config = new SparkConf().setAppName("ANNCLUSTERD").set("spark.locality.wait", "0")
.set("spark.submit.deployMode", "cluster").set("spark.driver.maxResultSize", "6g")
.set("spark.executor.memory", "6g").setMaster("spark://34.80.87.222:7077").set("spark.cores.max", "8")
.set("spark.blockManager.port", "10025").set("spark.driver.blockManager.port", "10026")
.set("spark.driver.port", "10027").set("spark.shuffle.service.enabled", "false")
.set("spark.dynamicAllocation.enabled", "false");
and run the same spark-submit command I am getting this error:
22/11/15 08:45:37 ERROR org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
22/11/15 08:45:37 WARN org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend: Application ID is not initialized yet.
22/11/15 08:45:37 INFO org.spark_project.jetty.server.AbstractConnector: Stopped Spark@40147317{HTTP/1.1, (http/1.1)}{0.0.0.0:0}
22/11/15 08:45:37 WARN org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
22/11/15 08:45:38 ERROR org.apache.spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:517)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:939)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:948)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:281)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:517)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:939)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:948)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Case 3: When I try to run the same JAR file in YARN cluster mode with 4 executors changing the config file to:
SparkConf config=new SparkConf().setAppName("ANN-SCL");
and the submit command I get this following error:
spark-submit --class main.GraphNetworkSCLAlgorithm --master yarn --deploy-mode cluster --num-executors 4 /home/aavashbhandari/dataset/RunSCL.jar /home/aavashbhandari/dataset/California_Nodes.txt /home/aavashbhandari/dataset/California_Edges.txt /home/aavashbhandari/dataset/California_part_4.txt 4 10000 10000 1
I get the following error, while the job gets accepted it never passes to the running stage.
Application application_1668488196607_0004 failed 2 times due to AM Container for appattempt_1668488196607_0004_000002 exited with exitCode: 13
Failing this attempt.Diagnostics: [2022-11-15 08:16:01.044]Exception from container-launch.
Container id: container_1668488196607_0004_02_000001
Exit code: 13
[2022-11-15 08:16:01.086]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at java.io.FileReader.<init>(FileReader.java:58)
at framework.UtilsManagement.readEdgeTxtFileReturnGraph(UtilsManagement.java:382)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:91)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
java.io.FileNotFoundException: /home/aavashbhandari/dataset/California_Edges.txt (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at java.io.FileReader.<init>(FileReader.java:58)
at edu.ufl.cise.bsmock.graph.YenGraph.readFromFile(YenGraph.java:171)
at edu.ufl.cise.bsmock.graph.YenGraph.<init>(YenGraph.java:21)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
java.io.FileNotFoundException: /home/aavashbhandari/dataset/California_Nodes.txt (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at java.io.FileReader.<init>(FileReader.java:58)
at framework.UtilsManagement.readTxtNodeFile(UtilsManagement.java:210)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
22/11/15 08:16:00 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: Uncaught exception:
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263)
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:220)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:470)
at org.apache.spark.deploy.yarn.ApplicationMaster.runImpl(ApplicationMaster.scala:305)
at org.apache.spark.deploy.yarn.ApplicationMaster.$anonfun$run$1(ApplicationMaster.scala:245)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:781)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1938)
at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:780)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:245)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:805)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
[2022-11-15 08:16:01.087]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at java.io.FileReader.<init>(FileReader.java:58)
at framework.UtilsManagement.readEdgeTxtFileReturnGraph(UtilsManagement.java:382)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:91)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
java.io.FileNotFoundException: /home/aavashbhandari/dataset/California_Edges.txt (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at java.io.FileReader.<init>(FileReader.java:58)
at edu.ufl.cise.bsmock.graph.YenGraph.readFromFile(YenGraph.java:171)
at edu.ufl.cise.bsmock.graph.YenGraph.<init>(YenGraph.java:21)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
java.io.FileNotFoundException: /home/aavashbhandari/dataset/California_Nodes.txt (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at java.io.FileReader.<init>(FileReader.java:58)
at framework.UtilsManagement.readTxtNodeFile(UtilsManagement.java:210)
at main.GraphNetworkSCLAlgorithm.main(GraphNetworkSCLAlgorithm.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
22/11/15 08:16:00 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: Uncaught exception:
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263)
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:220)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:470)
at org.apache.spark.deploy.yarn.ApplicationMaster.runImpl(ApplicationMaster.scala:305)
at org.apache.spark.deploy.yarn.ApplicationMaster.$anonfun$run$1(ApplicationMaster.scala:245)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:781)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1938)
at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:780)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:245)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:805)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
For more detailed output, check the application tracking page: http://spark-cluster-m:8188/applicationhistory/app/application_1668488196607_0004 Then click on links to logs of each attempt.
. Failing the application.
I am using the absolute file path to pass the file as an argument and but still running on the error. I am worried that my configurations are not correct
Upvotes: 0
Views: 735
Answers (1)
OneCricketeer
Reputation: 191953
Your file needs to exist on every executor, in the same location. Local mode obviously is only running on one machine... You can see in the Spark UI which node the code runs on, then you see FileNotFoundException in the logs, so then you can debug on your own by SSH to that machine and look around the file path to see if the exception is true...
You should upload the file to HDFS, not use local Unix file paths, or have Spark do it with the --files argument to spark-submit
Upvotes: 1
Related Questions
- Google cloud dataproc --files is not working
- DataProc Cluster Spark Job submission fails to start NodeManager
- Error while running via spark-submit while connecting to google BQ
- Spark on Google's Dataproc failed due to java.io.FileNotFoundException: /hadoop/yarn/nm-local-dir/usercache/root/appcache/
- Spark on Dataproc fails with java.io.FileNotFoundException:
- GCP Dataproc spark.jar.packages issue downloading dependencies
- Spark streaming on dataproc throws FileNotFoundException
- Spark job became incompatible with Google Dataproc
- Running app jar file on spark-submit in a google dataproc cluster instance
- Google Cloud Dataproc - Submit Spark Jobs Via Spark