CODEWITHSUNDEEP
amazon-dynamodbetlaws-glue
Jie Zhao
Jie Zhao

Reputation: 1

DynamoDb cross-account migration using Glue Job, GSI value cannot be null

When I'm using Glue Job for cross-account migration in DynamoDB, I need to transform the PK and GSI values by adding a fixed prefix like 'XXX'. However, if the gsi0_pk value from the source table is empty, it throws an error since GSI values can't be null. Amazon DynamoDB's global secondary indexes can be NULL through normal business process writes,I just don't understand why I can't write NULL values after these two conversion steps

source_df = dyf.toDF()
target_dynamic_frame = DynamicFrame.fromDF(source_df, glue_context, ‘target_df’)

Is there a way to skip or drop the gsi0_pk and gsi0_sk fields when they're empty? If they're not empty, I want to use the concat function to combine the partition key value with the prefix. I really don't want to add a placeholder default value for gsi0_pk.

Upvotes: 0

Views: 25

Answers (1)

Leeroy Hannigan
Leeroy Hannigan

Reputation: 19883

GSI keys can be left out, but cannot be Null, Glue is setting empty values to Null which is why you get the exception.

You can use DropNullFields class to drop any value that is empty, instead of converting it to Null:

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-transforms-DropNullFields.html

Upvotes: 0

Related Questions