CODEWITHSUNDEEP
pythonapache-sparkpyspark
Liangju Zeng
Liangju Zeng

Reputation: 343

How to create dataframe from list in Spark SQL?

Spark version : 2.1

For example, in pyspark, i create a list

test_list = [['Hello', 'world'], ['I', 'am', 'fine']]

then how to create a dataframe form the test_list, where the dataframe's type is like below:

DataFrame[words: array<string>]

Upvotes: 33

Views: 106865

Answers (5)

Krishna Tapse
Krishna Tapse

Reputation: 21

If Columns are in different list then use below code as per requirement

l1 =[1,2,3,4]
l2 =['a','b','c','d']
l3=[]
if len(l1)==len(l2):
    for i in range(len(l1)):
        l3.append((l1[i],l2[i]))
print('List:',l3)
column=['Id','Name']
df = spark.createDataFrame(l3,column)
display(df)
**Output:**
List:[(1, 'a'), (2, 'b'), (3, 'c'), (4, 'd')]
Id  Name
1   a
2   b
3   c
4   d

Upvotes: 0

hamza tuna
hamza tuna

Reputation: 1497

You should use list of Row objects([Row]) to create data frame.

from pyspark.sql import Row

spark.createDataFrame(list(map(lambda x: Row(words=x), test_list)))

Upvotes: 12

Grant Shannon
Grant Shannon

Reputation: 5055

i had to work with multiple columns and types - the example below has one string column and one integer column. A slight adjustment to Pushkr's code (above) gives:

from pyspark.sql.types import *

cSchema = StructType([StructField("Words", StringType())\
                      ,StructField("total", IntegerType())])

test_list = [['Hello', 1], ['I am fine', 3]]

df = spark.createDataFrame(test_list,schema=cSchema)

output:

 df.show()
 +---------+-----+
|    Words|total|
+---------+-----+
|    Hello|    1|
|I am fine|    3|
+---------+-----+

Upvotes: 25

Pushkr
Pushkr

Reputation: 3619

here is how - 

from pyspark.sql.types import *

cSchema = StructType([StructField("WordList", ArrayType(StringType()))])

# notice extra square brackets around each element of list 
test_list = [['Hello', 'world']], [['I', 'am', 'fine']]

df = spark.createDataFrame(test_list,schema=cSchema)

Upvotes: 39

Raju Bairishetti
Raju Bairishetti

Reputation: 354

   You can create a RDD first from the input and then convert to dataframe from the constructed RDD
   <code>  
     import sqlContext.implicits._
       val testList = Array(Array("Hello", "world"), Array("I", "am", "fine"))
       // CREATE RDD
       val testListRDD = sc.parallelize(testList)
     val flatTestListRDD = testListRDD.flatMap(entry => entry)
     // COnvert RDD to DF 
     val testListDF = flatTestListRDD.toDF
     testListDF.show
    </code>

Upvotes: -3

Related Questions