CODEWITHSUNDEEP
pysparksnowflake-cloud-data-platformaws-gluesnowflake-schemaaws-glue-spark
Lavish Patodi
Lavish Patodi

Reputation: 21

Glue PySpark Job: An error occurred while calling o100.pyWriteDynamicFrame

I am building data pipeline for migrating data from S3 bucket to Snowflake via AWS Glue by creating custom connector in AWS Glue.

I am getting below Error when running glue job:

**An error occurred while calling o100.pyWriteDynamicFrame. Glue ETL Marketplace: JDBC Data Sink: Column "serial_number" not found in schema Some(StructType(StructField(SERIAL_NUMBER,DoubleType,true), StructField(COMPANY_NAME,StringType,true), StructField(EMPLOYEE_MARKME,StringType,true), StructField(DESCRIPTION,StringType,true), StructField(LEAVE,LongType,true)))**

Note : Using below version's for the connector and driver respectively -

  1. Snowflake Spark connector - spark-snowflake_2.11-2.4.14-spark_2.3.jar
  2. Snowflake JDBC driver - snowflake-jdbc-3.8.3.jar
  3. S3 bucket in the same region as Glue.

Please let me know if anything I am missing or I am using incorrect version/unsupportable version of connector and driver........

Upvotes: 2

Views: 12864

Answers (2)

julibigdata
julibigdata

Reputation: 1

try to update the glue version to 4.0 or 5.0!

Upvotes: 0

Vivek Pant
Vivek Pant

Reputation: 21

Are you passing the column in double quotes? If yes, that might be the root cause. Snowflake implicitly converts all names to upper case unless you specify it in double quotes and it works the other way round too (if you have a column name called SERIAL_NUMBER and you select "serial_number" in your query, it will not be able to find that column.

Upvotes: 2

Related Questions