CODEWITHSUNDEEP
pysparkanalyticsaws-glue
user19195895
user19195895

Reputation: 41

AWS Glue job throwing Null pointer exception when writing df

I am trying to write a job to read data from S3 and write to BQ db (using connector), running the same script for other tables and it is working correctly, but for one of the tables the write is not working. It is working on the first run, but after first load the incremental runs throws this null pointer exception error. I have bookmarks enabled to fetch new data added in S3 and write to BQ database.

I am already handling the new data check, if there are files to process then proceed else abort job. In the job logs df is printing and count is printing too, everything seems to be working but as it runs the write df command the job fails. I am not sure what is the cause. Had tried to make the nullability of source and target to be same too, by setting the nullable property of source to True same as target, but it still fails.

Unable to understand the null pointer exception thrown.

Error: Caused by: java.lang.NullPointerException at com.google.cloud.bigquery.connector.common.BigQueryClient.loadDataIntoTable(BigQueryClient.java:532) at com.google.cloud.spark.bigquery.BigQueryWriteHelper.loadDataToBigQuery(BigQueryWriteHelper.scala:87) at com.google.cloud.spark.bigquery.BigQueryWriteHelper.writeDataFrameToBigQuery(BigQueryWriteHelper.scala:66) ... 42 more

Upvotes: 1

Views: 765

Answers (1)

user19195895
user19195895

Reputation: 41

The BQ connector by AWS had a bug. This was resolved when I contacted the AWS team and they suggested to use previous version of the connector.

So, using previous version of connector helped me resolve the issue.

Upvotes: 1

Related Questions