CODEWITHSUNDEEP
amazon-web-servicespysparkaws-glue
t_warsop
t_warsop

Reputation: 1270

AWS Glue NameError: name 'DynamicFrame' is not defined

I'm trying to convert a dataframe to a Dynamic Frame using the toDF and fromDF functions (https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame.html#aws-glue-api-crawler-pyspark-extensions-dynamic-frame-fromDF) as per the below code snippet:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
## @type: DataSource
## @args: [database = "test-3", table_name = "test", transformation_ctx = "datasource0"]
## @return: datasource0
## @inputs: []
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "test-3", table_name = "test", transformation_ctx = "datasource0")


foo = datasource0.toDF()
bar = DynamicFrame.fromDF(foo, glueContext, "bar")

However, I'm getting an error on the line:

bar = DynamicFrame.fromDF(foo, glueContext, "bar")

The error says:

NameError: name 'DynamicFrame' is not defined

I've tried the usual googling to no avail, I can't see what I've done wrong from other examples. Does anyone know why I'm getting this error and how to resolve it?

Upvotes: 4

Views: 14120

Answers (2)

Harsh Bafna
Harsh Bafna

Reputation: 2224

You need to import the DynamicFrame class from awsglue.dynamicframe module:

from awsglue.dynamicframe import DynamicFrame

There are lot of things missing in the examples provided with the AWS Glue ETL documentation.

However, you can refer to the following GitHub repository which contains lots of examples for performing basic tasks with Glue ETL:

AWS Glue samples

Upvotes: 2

Sandeep Fatangare
Sandeep Fatangare

Reputation: 2144

from awsglue.dynamicframe import DynamicFrame

Import DynamicFrame

Upvotes: 17

Related Questions