CODEWITHSUNDEEP
amazon-web-servicesapache-sparkaws-glue
Somasundaram Sekar
Somasundaram Sekar

Reputation: 5524

No provision to convert Spark DataFrame to AWS Glue DynamicFrame in scala

There is no equivalent of the below code to convert from Spark DataFrame to Glue DynamicFrame, is it intentional, what is the workaround?

# Convert to a dataframe and partition based on "partition_col"
partitioned_dataframe = datasource0.toDF().repartition(1)

# Convert back to a DynamicFrame for further processing.
partitioned_dynamicframe = DynamicFrame.fromDF(partitioned_dataframe, glueContext, "partitioned_df")

Upvotes: 0

Views: 475

Answers (1)

Yuriy Bondaruk
Yuriy Bondaruk

Reputation: 4750

Use this:

# Convert Spark's DataFrame to Glue's DynamicFrame
val partitioned_dynamicframe = DynamicFrame(partitioned_dataframe, glueContext)

Upvotes: 2

Related Questions