CODEWITHSUNDEEP
apache-sparkcluster-computingperformance-testing
Hawk360
Hawk360

Reputation: 171

Failed to start a cluster manually in Spark

I have just started using Spark Apache. I tested it in my local computer (Windows 10, Intel Core i5, 8 GB RAM) and everything worked correctly. When I tried to start a cluster manually I got a error as in the below attached image: enter image description here

Info from log (C:\Spark\logs\spark--org.apache.spark.deploy.master.Master-1-XXXXXX.out:

Spark Command: C:\Program Files\Java\jdk1.8.0_72\bin\java -cp C:\Spark/conf\; C:\Spark/lib/spark-assembly-1.6.0-hadoop2.6.0.jar;C:\Spark\lib\datanucleus-api-jdo-3.2.6.jar; C:\Spark\lib\datanucleus-core-3.2.10.jar;C:\Spark\lib\datanucleus-rdbms-3.2.9.jar -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip XXXXXX --port 7077 --webui-port 8080

I use these sources to resolve this issue but no success:
Spark Standalone Mode
How to Setup Local Standalone Spark Node
Setup a Apache Spark cluster in your single standalone machine

Thank you for any feedback.

Upvotes: 0

Views: 579

Answers (1)

avloss
avloss

Reputation: 2626

I think you're slightly mixed up. When running Spark on Windows machine - it would make sense to either use remote master, or to run it locally. In both cases - use bin\spark-shell.cmd (or another bin/*.cmd). You should only execute commands which end with .cmd on Windows machine. When you run start-master.sh what you saying is - I want to use this machine as a resource manager of a spark cluster, so other spark clients can connect to it and execute queries. This is probably NOT what you want. When you're doing local[*], you're already using all the local resources, there's no point to start "master". If you need more resources - start a remote cluster (for instance EMR) and then connect to it with bin\spark-shell.cmd --master AMAZON.IP:7077

Upvotes: 1

Related Questions