CODEWITHSUNDEEP
jsonscalaapache-sparkapache-spark-sql
justanothertekguy
justanothertekguy

Reputation: 183

Get the value from the JSON string

I have a dataframe and I am trying to convert the data to JSON so that I can send a message.

val df = Seq((123456, 40, "ABCD", "DEFG")).toDF("ID", "RunId", "Val1", "Val2")
val msgDf = df.select(struct("ID", "RunId", "Val1", "Val2").alias("message"))
msgDf.toJSON.show(false)

Schema for msgDf

message:struct
      ID:integer
      RunId:integer
      Val1:string
      Val2:string

Output I am getting is

+----------------------------------------------------------------+
|value                                                           |
+----------------------------------------------------------------+
|{"message":{"ID":123456,"RunId":40,"Val1":"ABCD","Val2":"DEFG"}}|
+----------------------------------------------------------------+

I just want to capture the below and send it as messgage

{"ID":123456,"RunId":40,"Val1":"ABCD","Val2":"DEFG"}

How can I achive this?

Upvotes: 0

Views: 48

Answers (2)

koiralo
koiralo

Reputation: 23119

You can use to_json method as

val df = Seq((123456, 40, "ABCD", "DEFG"))
  .toDF("ID", "RunId", "Val1", "Val2")

val msgDf = df.select(to_json(struct("ID", "RunId", "Val1", "Val2")).as("message"))

Or simply

df.select(to_json(struct(df.columns.map(col): _*)).alias("message"))

msgDf.show(false)

Or

Seq((123456, 40, "ABCD", "DEFG")).toDF("ID", "RunId", "Val1", "Val2")
  .toJSON.show(false)

Result:

+----------------------------------------------------+
|message                                             |
+----------------------------------------------------+
|{"ID":123456,"RunId":40,"Val1":"ABCD","Val2":"DEFG"}|
+----------------------------------------------------+

Upvotes: 1

mck
mck

Reputation: 42422

No need to use struct - doing toJSON directly will do the job:

val df = Seq((123456, 40, "ABCD", "DEFG")).toDF("ID", "RunId", "Val1", "Val2")

df.toJSON.show(false)
+----------------------------------------------------+
|value                                               |
+----------------------------------------------------+
|{"ID":123456,"RunId":40,"Val1":"ABCD","Val2":"DEFG"}|
+----------------------------------------------------+

Upvotes: 1

Related Questions