CODEWITHSUNDEEP
apache-sparkapache-spark-sqlapache-spark-mllibapache-spark-mlapache-spark-2.0
Almog Gelber
Almog Gelber

Reputation: 31

Adding custom metadata to DataFrame schema using iceberg table format

I'm adding custom metadata into the DataFrames schema in my PySpark application using StructField's metadata field

It worked fine when I wrote parquet files directly into s3. The custom metadata was available when reading these parquet files as expected.

But it's not working using iceberg table format. There is no error, but the df.schema.fields.metadata is always empty.

Is there a way to solve it?

Upvotes: 1

Views: 673

Answers (1)

Almog Gelber
Almog Gelber

Reputation: 31

Solved by making sure the key is always 'comment'

For example: {'comment': 'my_metadata_info_field'}

Upvotes: 2

Related Questions