CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sqlbase64
Data_101
Data_101

Reputation: 953

Decode Base64 within Spark Dataframe

key value topic partition
null dGVzdF90ZXh0 Topic.Name 0
null dGVzdF90ZXh0 Topic.Name 0
null dGVzdF90ZXh0 Topic.Name 0

df_1

I have a Spark dataframe where the column value is a Base64 encoded. I would like to be able to add a column at the end of the dataframe with an unencoded version of it.

import base64

df_2 = df_1.withColumn('unencoded_base64',base64.b64decode(df_1.value))

The above code gives me the error:

TypeError: a2b_base64() argument 1 must be convertible to a buffer, not Column

Upvotes: 5

Views: 24341

Answers (2)

s510
s510

Reputation: 2822

adding to Remesh's answer, you need to cast it to string to get it in "readable" context -

df_2=df_1.withColumn('unencoded_base64',unbase64(df_1.value).cast("string"))

Upvotes: 7

Ramesh Maharjan
Ramesh Maharjan

Reputation: 41957

You can use unbase64 inbuilt function for that 

from pyspark.sql.functions import unbase64
df_2 = df_1.withColumn('unencoded_base64', unbase64(df_1.value))

which should give you 

+----+------------+----------+---------+----------------------------+
|key |value       |topic     |partition|unencoded_base64            |
+----+------------+----------+---------+----------------------------+
|null|dGVzdF90ZXh0|Topic.Name|0        |[74 65 73 74 5F 74 65 78 74]|
|null|dGVzdF90ZXh0|Topic.Name|0        |[74 65 73 74 5F 74 65 78 74]|
|null|dGVzdF90ZXh0|Topic.Name|0        |[74 65 73 74 5F 74 65 78 74]|
+----+------------+----------+---------+----------------------------+

I hope the answer is helpful

Upvotes: 11

Related Questions