SeaTea
Reputation: 51
PySpark - Numpy Not Found in Cluster Mode - ModuleNotFoundError
I'm running a job on a PySpark cluster for the first time. It runs perfectly in standalone mode on the name node. However, when it runs in the cluster:
spark-submit --master yarn \
--deploy-mode client \
--driver-memory 6g \
--executor-memory 6g \
--executor-cores 2 \
--num-executors 10 \
nearest_neighbor.py
It begins complaining that numpy isn't installed:
from pyspark.ml.param.shared import *
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0024/container_1582692915671_0024_01_000002/pyspark.zip/pyspark/ml/param/__init__.py", line 26, in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
However, that module is confirmed to be installed on all nodes of the cluster (using pip3 install numpy). I've also confirmed that the job is running in python3:
sys.version_info(major=3, minor=6, micro=9, releaselevel='final', serial=0)
If comment out the call in my script to 'import numpy as np', it still complains about numpy not being installed, so I began to suspect that something besides my script was trying to import the module improperly.
By commenting out code in the script, I discovered there are some functions I'm calling that must be trying to call a different version of numpy on the back end. Specifically, this call throws the error about numpy not being installed:
isNonZeroVector = udf(lambda x: x.numNonzeros() > 0, BooleanType())
trans_corpus_df = trans_corpus_df.select('id', 'features', \
isNonZeroVector('features').alias('non_zero'))
trans_corpus_df = trans_corpus_df.filter(trans_corpus_df.non_zero == True)
It appears that Vector has methods (in this case numNonzeroes()) that is based on Numpy, and somehow it's failing to find the numpy module.
I confirmed that Python2 doesn't have numpy installed:
Python 2.7.15+ (default, Oct 7 2019, 17:39:04)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named numpy
So I installed the Python2 version of numpy on all the nodes. This didn't change the error at all, unfortunately.
I added a line to check where numpy is (print(np.file)) and it gives this:
/home/ubuntu/.local/lib/python3.6/site-packages/numpy/__init__.py
The permissions all the way down allow Ubuntu to access that directory, so I don't think it's a permissions issue.
Any tips are greatly appreciated!
Full error report:
20/02/26 14:21:19 ERROR TaskSetManager: Task 0 in stage 6.0 failed 4 times; aborting job
Traceback (most recent call last):
File "/home/ubuntu/server/nearest_neighbor.py", line 243, in <module>
main(args)
File "/home/ubuntu/server/nearest_neighbor.py", line 209, in main
hash_model = mh.fit(trans_corpus_df)
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/base.py", line 132, in fit
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 295, in _fit
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 292, in _fit_java
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
File "/home/ubuntu/server/spark-2.4.4-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o160.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure: Lost task 0.3 in stage 6.0 (TID 28, ip-172-31-5-228.ec2.internal, executor 3): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/worker.py", line 377, in main
process()
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/worker.py", line 372, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 345, in dump_stream
self.serializer.dump_stream(self._batched(iterator), stream)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 141, in dump_stream
for obj in iterator:
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 334, in _batched
for item in iterator:
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 147, in load_stream
yield self._read_with_length(stream)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 172, in _read_with_length
return self.loads(obj)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 580, in loads
return pickle.loads(obj, encoding=encoding)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 869, in _parse_datatype_json_string
return _parse_datatype_json_value(json.loads(json_string))
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 886, in _parse_datatype_json_value
return _all_complex_types[tpe].fromJson(json_value)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 577, in fromJson
return StructType([StructField.fromJson(f) for f in json["fields"]])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 577, in <listcomp>
return StructType([StructField.fromJson(f) for f in json["fields"]])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 434, in fromJson
_parse_datatype_json_value(json["type"]),
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 888, in _parse_datatype_json_value
return UserDefinedType.fromJson(json_value)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 736, in fromJson
m = __import__(pyModule, globals(), locals(), [pyClass])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/__init__.py", line 22, in <module>
from pyspark.ml.base import Estimator, Model, Transformer, UnaryTransformer
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/base.py", line 24, in <module>
from pyspark.ml.param.shared import *
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/param/__init__.py", line 26, in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:456)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:410)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:365)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3389)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2550)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2550)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2550)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2557)
at org.apache.spark.ml.feature.LSH.fit(LSH.scala:328)
at org.apache.spark.ml.feature.LSH.fit(LSH.scala:304)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/worker.py", line 377, in main
process()
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/worker.py", line 372, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 345, in dump_stream
self.serializer.dump_stream(self._batched(iterator), stream)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 141, in dump_stream
for obj in iterator:
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 334, in _batched
for item in iterator:
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 147, in load_stream
yield self._read_with_length(stream)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 172, in _read_with_length
return self.loads(obj)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/serializers.py", line 580, in loads
return pickle.loads(obj, encoding=encoding)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 869, in _parse_datatype_json_string
return _parse_datatype_json_value(json.loads(json_string))
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 886, in _parse_datatype_json_value
return _all_complex_types[tpe].fromJson(json_value)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 577, in fromJson
return StructType([StructField.fromJson(f) for f in json["fields"]])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 577, in <listcomp>
return StructType([StructField.fromJson(f) for f in json["fields"]])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 434, in fromJson
_parse_datatype_json_value(json["type"]),
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 888, in _parse_datatype_json_value
return UserDefinedType.fromJson(json_value)
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/sql/types.py", line 736, in fromJson
m = __import__(pyModule, globals(), locals(), [pyClass])
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/__init__.py", line 22, in <module>
from pyspark.ml.base import Estimator, Model, Transformer, UnaryTransformer
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/base.py", line 24, in <module>
from pyspark.ml.param.shared import *
File "/tmp/hadoop-ubuntu/nm-local-dir/usercache/ubuntu/appcache/application_1582692915671_0010/container_1582692915671_0010_01_000004/pyspark.zip/pyspark/ml/param/__init__.py", line 26, in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:456)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:410)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Upvotes: 1
Views: 5040
Answers (2)
Sajad Safarveisi
Reputation: 101
If you are using pyspark via Jupyter notebook, you can use the following script (before you create a spark session) to set an environment variable for the pyspark python.
os.environ['PYSPARK_PYTHON'] = ".../bin/python"
Where the string on the right is the path to python (installed on your local system (node) and not the one for pyspark). Note that the Numpy package should already be installed for the script to work.
This is not directly related to your question. However, I put it here for future observers of this issue.
Upvotes: 2
SeaTea
Reputation: 51
This ended up solving the problem. I ended up adding the following lines to .bashrc in all nodes of the cluster:
export PYTHONPATH=/usr/bin/python3:/home/ubuntu/.local/lib/python3.6/site-packages:$PYTHONPATH
export PYSPARK_PYTHON=/usr/bin/python3:/home/ubuntu/.local/lib/python3.6/site-packages:$PYSPARK_PYTHON
Where the first is the path to python3 and the second is the path to the installed packages.
Upvotes: 2
Related Questions
- Spark can't find Python module
- PySpark: ModuleNotFoundError: No module named 'app'
- ImportError: No module named numpy on spark workers
- Cannot use numpy with Spark
- No module named numpy when spark-submitting
- Spark No module named found
- pyspark: ImportError: No module named numpy
- Pyspark module not found
- NumPy exception when using MLlib even though Numpy is installed
- Installing Modules for SPARK on worker nodes