CODEWITHSUNDEEP
amazon-web-servicesaws-glueaws-glue-data-catalog
Rahul Berry
Rahul Berry

Reputation: 57

How to avoid use of crawler in aws glue

AWS glue crawler has cost associated with it, how to avoid us of the crawler in aws glue.

Is there any way we can avoid the use of crawler and infer schema from any other option, so that cost can be reduced.

Upvotes: 0

Views: 1360

Answers (2)

Adam Luchjenbroers
Adam Luchjenbroers

Reputation: 5019

In addition to what bdcloud has said, it's also possible to add tables to the data catalogue using the 'AWS::Glue::Table' resource in CloudFormation.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-glue-table.html

It's easier to do this if you have a table schema you can use as a template (aws glue get-table --database-name <db name> --name <table name> will give you JSON that is pretty close to what CloudFormation is expecting).

Again, you need to know your schema in advance, but choose the approach that best fits the workflow you're going with.

Upvotes: 1

Prabhakar Reddy
Prabhakar Reddy

Reputation: 5124

You can use Athena to create tables in Glue catalog, but to do so you need to know the schema of the file or you can get the DDL from the existing table created by running SHOW CREATE TABLE <table-name> in Athena and then you can modified the DDL statement according to your schema.

DDL queries are free in Athena and incurs no charges.

One other way of doing it is by issuing a Glue create table API call. Please refer to this for python syntax.

Upvotes: 0

Related Questions