CODEWITHSUNDEEP
pysparkapache-spark-sql
user9226665
user9226665

Reputation: 103

List to DataFrame in pyspark

Can someone tell me how to convert a list containing strings to a Dataframe in pyspark. I am using python 3.6 with spark 2.2.1. I am just started learning spark environment and my data looks like below

my_data =[['apple','ball','ballon'],['cat','camel','james'],['none','focus','cake']]

Now, i want to create a Dataframe as follows

---------------------------------
|ID | words                     |
---------------------------------
 1  | ['apple','ball','ballon'] |
 2  | ['cat','camel','james']   |

I even want to add ID column which is not associated in the data

Upvotes: 4

Views: 27823

Answers (4)

Mohana B C
Mohana B C

Reputation: 5487

Simple Approach:

my_data =[['apple','ball','ballon'],['cat','camel','james'],['none','focus','cake']]

spark.sparkContext.parallelize(my_data).zipWithIndex() \
toDF(["id", "words"]).show(truncate=False)

+---------------------+-----+
|id                   |words|
+---------------------+-----+
|[apple, ball, ballon]|0    |
|[cat, camel, james]  |1    |
|[none, focus, cake]  |2    |
+---------------------+-----+

Upvotes: 0

Jie
Jie

Reputation: 1284

Try this -- the simplest approach 

  from pyspark.sql import *
  x = Row(utc_timestamp=utc, routine='routine name', message='your message')
  data = [x]
  df = sqlContext.createDataFrame(data)

Upvotes: 0

abhjt
abhjt

Reputation: 442

Try this -

data_array = []
for i in range (0,len(my_data)) :
    data_array.extend([(i, my_data[i])])

df = spark.createDataframe(data = data_array, schema = ["ID", "words"])

df.show()

Upvotes: 0

akuiper
akuiper

Reputation: 215117

You can convert the list to a list of Row objects, then use spark.createDataFrame which will infer the schema from your data:

from pyspark.sql import Row
R = Row('ID', 'words')

# use enumerate to add the ID column
spark.createDataFrame([R(i, x) for i, x in enumerate(my_data)]).show() 
+---+--------------------+
| ID|               words|
+---+--------------------+
|  0|[apple, ball, bal...|
|  1| [cat, camel, james]|
|  2| [none, focus, cake]|
+---+--------------------+

Upvotes: 8

Related Questions