CODEWITHSUNDEEP
aws-gluepyspark
user-2147482338
user-2147482338

Reputation: 85

Convert pyspark dataframe to dynamic dataframe

I have a pyspark dataframe. I was able to convert dynamic dataframe to spark dataframe by persons.toDF(). I want to convert the spark dataframe again back to dynamic dataframe in pyspark.I wanted to cast my column to timestamp and again convert it to dynamic dataframe to resolveChoices. Please help me

Upvotes: 3

Views: 10767

Answers (2)

Sergii Sholovii
Sergii Sholovii

Reputation: 61

at least you need pyspark.context, awsglue.context and awsglue.dynamicframe There is example :

from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame

sc = SparkContext()
glueContext = GlueContext(sc)

NewDynamicFrame = DynamicFrame.fromDF(persons, glueContext, "nested")

"persons" is your DataFrame

Please check following links :

  1. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-samples-medicaid.html

  2. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame.html#aws-glue-api-crawler-pyspark-extensions-dynamic-frame-fromDF

Upvotes: 4

Harsh Bafna
Harsh Bafna

Reputation: 2224

You can create a dynamic frame from dataframe using the fromDF function.

Basic Syntax

dyf = fromDF(dataframe, glue_ctx, name)

where,

  • dataframe – The Apache Spark SQL DataFrame to convert (required).

  • glue_ctx – The GlueContext Class object that specifies the context for this transform (required).

  • name – The name of the resulting DynamicFrame (required).

Reference : Dynamic frame from dataframe

Upvotes: 3

Related Questions