CODEWITHSUNDEEP
apache-sparkpyspark
rosefun
rosefun

Reputation: 1857

Spark: AttributeError: 'SQLContext' object has no attribute 'createDataFrame'

How to createDataFrame from a dict? I use the following code and meet errors.

from pyspark import SparkContext, SQLContext
sc = SparkContext.getOrCreate()
spark = SQLContext(sc)

result_dict = {'a':3,'b':44}
data = list(map(list, result_dict.items()))
f_rdd = spark.createDataFrame(data, ["A", "B"]).repartition(1)

Error:

AttributeError                      Traceback (most recent call last)
<ipython-input-10-a25453caa1c3> in <module>
      5 result_dict = {'a':3,'b':44}
      6 data = list(map(list, result_dict.items()))
----> 7 f_rdd = spark.createDataFrame(data, ["A", "B"]).repartition(1)

AttributeError: 'SQLContext' object has no attribute 'createDataFrame'

Upvotes: 1

Views: 2673

Answers (1)

kites
kites

Reputation: 1405

You can try this way:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName('so')\
    .getOrCreate()

sc = spark.sparkContext

map = {'a': 3, 'b': 44}
data = sc.parallelize([(k, v) for k, v in map.items()]).toDF(['A', 'B'])

data.show()

# +---+---+
# |  A|  B|
# +---+---+
# |  a|  3|
# |  b| 44|
# +---+---+

Upvotes: 1

Related Questions