CODEWITHSUNDEEP
pythonpyspark
pyds_learner
pyds_learner

Reputation: 519

create pyspark dataframe based on condition and list of values

I have a value in a variable - ID as 1 and a list of ten values say

LIST1 = [1,2,3,4,5,6,7,8,9,10].

Now I wanted to create a pyspark data frame as below:

ID  LIST
1   1
1   2
1   3
1   4
1   5
1   6
1   7
1   8
1   9
1   10

NOTE: The List1 length is dynamic, based on its length we need to have the rows accordingly.

Upvotes: 2

Views: 797

Answers (1)

Geekmoss
Geekmoss

Reputation: 682

It depends on whether the ID is constant or you will even have List2 with ID 2 and then want to union for both into one DataFrame.

As far as the constant is concerned, there are two options:

ID = 1
LIST1 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

source = list(map(lambda x: (ID, x), LIST1))
# source: [(1, 1), (1, 2), (1, 3), (1, 4), (1, 5), (1, 6), (1, 7), (1, 8), (1, 9), (1, 10)]

df = spark.createDataFrame(source, ['ID', 'LIST'])
df.show()
# +---+----+                                                                      
# | ID|LIST|
# +---+----+
# |  1|   1|
# |  1|   2|
# |  1|   3|
# |  1|   4|
# |  1|   5|
# |  1|   6|
# |  1|   7|
# |  1|   8|
# |  1|   9|
# |  1|  10|
# +---+----+

or

from pyspark.sql.functions import lit

ID = 1
LIST1 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

source = list(map(lambda x: (x,), LIST1))
# createDataFrame needs iter of iters -> list/tuple of lists/tuples
df = spark.createDataFrame(source, ['LIST'])
df.withColumn('ID', lit(ID)).show()
+----+---+
|LIST| ID|
+----+---+
|   1|  1|
|   2|  1|
|   3|  1|
|   4|  1|
|   5|  1|
|   6|  1|
|   7|  1|
|   8|  1|
|   9|  1|
|  10|  1|
+----+---+

Upvotes: 2

Related Questions