CODEWITHSUNDEEP
javapythonapache-spark
Run
Run

Reputation: 57176

Spark + Python - Java gateway process exited before sending the driver its port number?

Why do I get this error on my browser screen,

: Java gateway process exited before sending the driver its port number args = ('Java gateway process exited before sending the driver its port number',) message = 'Java gateway process exited before sending the driver its port number'

for,

#!/Python27/python
print "Content-type: text/html; charset=utf-8"
print

# enable debugging
import cgitb
cgitb.enable()

import os
import sys

# Path for spark source folder
os.environ['SPARK_HOME'] = "C:\Apache\spark-1.4.1"

# Append pyspark to Python Path
sys.path.append("C:\Apache\spark-1.4.1\python")

from pyspark import SparkContext
from pyspark import SparkConf

print ("Successfully imported Spark Modules")

# Initialize SparkContext
sc = SparkContext('local')
words = sc.parallelize(["scala","java","hadoop","spark","akka"])
print words.count()

I followed this example.

Any ideas how I can fix it?

Upvotes: 7

Views: 24971

Answers (4)

Gus Cunha
Gus Cunha

Reputation: 11

I've been trying to solve this problem since yesterday. I've just solved it.

I did exactly what this link (https://sparkbyexamples.com/pyspark/pyspark-exception-java-gateway-process-exited-before-sending-the-driver-its-port-number/) tells us:

(1) Set PYSPARK_SUBMIT_ARGS;

(2) Install Open JDK;

(3) Set JAVA_HOME Environment Variable.

I won't try to explain these steps to you because the link can do it much better than me (you just need to follow the steps).

By the way, it is a good resource in case you're learning pyspark.

Upvotes: 1

user7210421
user7210421

Reputation: 21

My friend has met with the same problem as yours. I checked her computer and found that she had two versions of Java in it. I uninstalled the older one and rewrote the $JAVA_HOME value. The problem was solved.

Upvotes: 2

theheadofabroom
theheadofabroom

Reputation: 22019

I had a similar issue to this, and eventually when I looked at the my test's output there were error messages from $SPARK_HOME/bin/spark-class, with line numbers.

After tracking down what was going on on the affected lines it turned out that there were single quotes around the $JAVA_HOME value in my environmental variables, which was causing issues with path expansion (it was assumed for some reason to be relative to my home directory, rather than an absolute path)

While this may not be your exact issue, it is worth examining the start of your output for extra information to help in narrowing down the root cause.

Upvotes: 2

architectonic
architectonic

Reputation: 3129

Check if there are any extra information before the Error line that says:

Error: Could not create the Java Virtual Machine.

In my case it was an invalid option that I had set in the conf file. Memory (initial heap size) is not allowed to have a comma: 3.5g is for example not acceptable whereas 3500m is.

Upvotes: 2

Related Questions