CODEWITHSUNDEEP
jsonapache-sparkpysparkapache-spark-sql
user149635
user149635

Reputation: 65

Json file : create table or dataframe by using python pyspark?

I have a file that have this type of schema :

enter image description here

and trying to create a table or dataframe base on them. Before that, have managed to upload the file using this script :

first_row_is_header = "true"
infer_schema = "true"
delimiter = ":"
df4 = spark.read.option("multiLine","true") \
.option("inferSchema", infer_schema) \
.option("sep", delimiter) \
.json ("/FileStore/tables/Ca_sect2.json")

End result become like this :

enter image description here

How do I convert the result below into table form? Have tried several types of scripts like dataFrame.createOrReplaceTempView, but keep on failing to get the result. Grateful for some help, as have been trying for 2 days.

Thank you very much.

Upvotes: 0

Views: 797

Answers (1)

blackbishop
blackbishop

Reputation: 32720

What you want is to explode the array structure of developerSales: 

df.withColumn("sales", explode(col("developerSales")))\
  .select(col("developer"), col("sales.*"))\
  .show()

Upvotes: 1

Related Questions