CODEWITHSUNDEEP
apache-sparkapache-spark-standalone
opensam
opensam

Reputation: 368

How to use start-all.sh to start standalone Worker that uses different SPARK_HOME (than Master)?

I have installed spark 2.1.1 on 2 machines but in different relative locations ie in one machine I have installed somewhere on an NTFS drive and on the other one I have installed it on an ext4 drive. I am trying to start a cluster in standalone mode with 2 slaves and a master by having 1 Master and 1 slave on 1 machine and 1 slave on other machine.

When I try to start this cluster via start-all.sh script on master node, I get the following error :-

192.168.1.154: bash: line 0: cd: /home/<somePath>/spark-2.1.1-bin-hadoop2.7: No such file or directory

I have set proper SPARK_HOME in respective bashrc files. Below is my slave file (in the 1 master + 1 slave machine)

localhost
192.168.1.154

I can remotely login to the 1 slave machine via ssh. I am able to run Spark cluster individually in each machine.

It is my understanding when I try to remotely start a slave from my master machine via start-all.sh script it is trying to goto the location where spark is installed on master node, but as on slave node the spark is installed on a different location, it fails. Can anyone please tell me how can I rectify this problem?

Upvotes: 1

Views: 6895

Answers (3)

Chris Smith
Chris Smith

Reputation: 688

In my case I had 2 Macs and 1 PC/Linux machine as workers. 1 of the Macs acted as a master as well.

On the Macs, I had installed spark under /Users/<user>/spark and set my $SPARK_HOME to this path.

On the Linux machine (ubuntu), I had setup the spark directory under /home/<user>/spark. When running start-all.sh on my spark master machine (1 of the Macs) would cause an error on the Linux worker: 192.168.1.33: bash: line 1: cd: /Users/<user>/spark: No such file or directory 192.168.1.33: bash: line 1: /Users/<user>/spark/sbin/start-worker.sh: No such file or directory

In order to fix the pathing problem, I mimicked the Mac by creating a symbolic link on the Linux machine pointing a "/Users" directory to the "/home" directory. This tricked Spark into working on that Linux machine/worker just as it works on the Macs

cd /

sudo ln -s home Users

This is probably not the most elegant solution but it meant I did not need to maintain my own version of start-all.sh and its associated subscripts.

Upvotes: 0

user4710306
user4710306

Reputation:

You should check your ~/.bashr. You can see my bashrc below:

export JAVA_HOME=/usr/local/java/jdk1.8.0_121
export JRE_HOME=$JAVA_HOME/jre
export SCALA_HOME=/usr/local/src/scala/scala-2.12.1
export SPARK_HOME=/usr/local/spark/2.1.0
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin

Finally, you have to update your bashrc environment:

source ~/.bashrc

Upvotes: 0

Jacek Laskowski
Jacek Laskowski

Reputation: 74669

In start-all.sh you can find the following:

if [ -z "${SPARK_HOME}" ]; then
  export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
fi

# Load the Spark configuration
. "${SPARK_HOME}/sbin/spark-config.sh"

# Start Master
"${SPARK_HOME}/sbin"/start-master.sh

# Start Workers
"${SPARK_HOME}/sbin"/start-slaves.sh

which has nothing to do with the Spark installation on the standalone master. start-all.sh simply uses whatever SPARK_HOME you've defined globally and uses it across all nodes in the cluster, for standalone master and workers.

In your case, I'd recommend writing a custom startup script that would start the standalone Master and workers per respective SPARK_HOME env vars.

start-slaves.sh (source here) does simply the following:

cd "${SPARK_HOME}" \; "${SPARK_HOME}/sbin/start-slave.sh" "spark://$SPARK_MASTER_HOST:$SPARK_MASTER_PORT"

And so there is not much magic going on, but to ssh to every node and execute the command line.

I think I'd even use Ansible for this.

Upvotes: 3

Related Questions