CODEWITHSUNDEEP
amazon-web-servicesaws-glueamazon-athena
Stagg
Stagg

Reputation: 2855

Unable to access data in tables configured using Partition Projection from AWS glue jobs using create_dynamic_frame.from_catalog

I've setup a table in Athena using partition projection. I haven't defined any partitions in the glue metadata catalogue and I can view the data in Athena OK using SQL.

When I go and setup a Glue job using this table, Glue doesn't seem to have access to the data:

data = glueContext.create_dynamic_frame.from_catalog(
    database="db", table_name="a_table")

print (data.count()) # returns 0 :(

Is there any way to access the data without needing to define Glue metadata partitions? I was under the impression that if Athena could see the data, so could Glue.

Upvotes: 2

Views: 809

Answers (1)

Theo
Theo

Reputation: 132862

Glue does not support Partition Projection, it's an Athena-only feature.

Glue ETL uses Spark, and Athena is Presto under the hood (with modifications, including Partition Projection). Glue ETL also does not support Athena views, and various other minor things.

Upvotes: 6

Related Questions