CODEWITHSUNDEEP
pythonapache-sparkpysparkimporterrorapache-spark-sql
mflowww
mflowww

Reputation: 7465

ImportError: cannot import name sqlContext

I'm using pyspark to read some csv data to spark Dataframe.

I try to import pyspark module as follows:

from pyspark.sql import sqlContext

Why do I get the following error? How to fix it?

ImportError: cannot import name sqlContext

I'm using Python 2.7 and Spark 2.0.1

Upvotes: 4

Views: 8504

Answers (2)

Kranthi Kumar
Kranthi Kumar

Reputation: 4614

You can also try using just pyspark, I have the same issue in my Jupyter Notebook you can resolve using

from pyspark import SQLContext

Upvotes: 0

santon
santon

Reputation: 4625

It's probably because you don't have your python pat set correctly. I find the following function useful when configuring my python environments.

def configure_spark(spark_home=None, pyspark_python=None, conf_dir=None):
    """Configures the Python path for importing pyspark

    Sets the SPARK_HOME and PYSPARK_PYTHON environment variables and modifies
    the Python PATH so the pyspark package can be imported.

    Args:
        spark_home (str): Path of SPARK_HOME. Defaults to SPARK_HOME module
            variable.
        pyspark_python (str): Path to Python binary to use in PySpark. Defaults
            to the currently executing Python binary.
        conf_dir (str): Path to configuration directory
    """

    # Set the configuration directory with some basic sanity checks:
    if conf_dir:
        if not os.path.isdir(conf_dir):
            raise OSError("Spark config directory not found: %s" % conf_dir)

        expected_conf = {'spark-env.sh', 'spark-defaults.conf'}
        found_conf = expected_conf - set(os.listdir(conf_dir))
        if found_conf:
            warnings.warn("Some configuration files were not found: %s" % found_conf)

        os.environ['SPARK_CONF_DIR'] = conf_dir

    spark_home = spark_home or SPARK_HOME
    os.environ['SPARK_HOME'] = spark_home

    if not os.path.isdir(spark_home):
        raise OSError("Specified SPARK_HOME is not a valid directory: %s" % spark_home)

    # Add the PySpark directories to the Python path:
    libs = glob(os.path.join(spark_home, 'python', 'lib', '*.zip'))
    if len(libs) < 2:
        raise OSError("Pyspark libraries not found in %s" % spark_home)
    for lib in libs:
        sys.path.insert(1, lib)

    # If PYSPARK_PYTHON isn't specified, use currently running Python binary:
    pyspark_python = pyspark_python or sys.executable
    os.environ['PYSPARK_PYTHON'] = pyspark_python

Upvotes: 3

Related Questions