Reputation: 1867
When I submit a python file to yarn, I meet this error:
ERROR yarn.ApplicationMaster: User class threw exception: java.io.IOException: Cannot run program "jupyter": error=2, No such file or directory
java.io.IOException: Cannot run program "jupyter": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:98)
at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
I submit by :
spark-submit --master yarn \
--deploy-mode cluster \
--executor-memory 8g \
--driver-memory 10g \
--num-executors 100 \
--executor-cores 10 \
test.py
I try to set export PYSPARK_PYTHON=/opt/anaconda3/bin/python
in ~/.bashrc
, but meet another error: /opt/bin/python: No such file or directory
.
But I'm sure I the file /opt/bin/python
exists.
The .bashrc
files is as follows:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/opt/anaconda3/etc/profile.d/conda.sh" ]; then
. "/opt/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/opt/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
# Auto deploy profile
source /etc/profile
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64
export PATH=$JAVA_HOME/bin:$PATH
@
"~/.bashrc" 76L, 2780C
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export SPARK_HOME=/home//spark2
export PATH=$PATH:$SPARK_HOME/bin
# hadoop
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$JAVA_HOME/lib/tools.jar
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$JAVA_HOME/lib/dt.jar
# JAVA
#export CLASSPATH=$(JARS=("$LIB"/*.jar); IFS=:; echo "${JARS[*]}")
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib/tools.jar
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib/dt.jar
# Jupyter notebook for pyspark
export PYSPARK_PYTHON=/opt/anaconda3/bin/python
I can run my code on local spark successfully.
Any solution for it?
Upvotes: 0
Views: 1071
Reputation: 8678
I have faced similar problem. Following solution solve my issue
update ~/.bashrc as below
PYSPARK_PYTHON=/usr/bin/python3
Upvotes: 0