CODEWITHSUNDEEP
amazon-web-servicesaws-cloudformationaws-glue
Dimitar Egumenovski
Dimitar Egumenovski

Reputation: 17

How can I set "create a single schema for each s3 path" in cloudformation?

I want to create a crawler resource from CFN (Cloudformation).
Here is my code: 

Type: AWS::Glue::Crawler
    Properties: 
      Name: !Ref GlueCrawlerName
      Role: !GetAtt crawlerRole.Arn
      Description: AWS Glue crawler to crawl DLG data
      DatabaseName: !Ref GlueDatabaseName
      Targets:
        S3Targets:
          - Path: 
              !Join
              - ''
              - - 's3://'
                - !Ref s3bucket
                - '/'
                - !Ref GlueTableName
      SchemaChangePolicy:
        UpdateBehavior: UPDATE_IN_DATABASE
        DeleteBehavior: DEPRECATE_IN_DATABASE
      Schedule:
        ScheduleExpression: cron(0 1 * * ? 2019)

Everything is alright as expected, only 'Create a single schema for each S3 path' is false. Which property is for this to set to true?

Upvotes: 1

Views: 4196

Answers (2)

Sandeep Fatangare
Sandeep Fatangare

Reputation: 2144

Do you need one table per subfolder or only one table at the root level of the s3 path?

for single root level table, append following in your CFN:

Configuration: "{\"Version\":1.0,\"Grouping\":{\"TableGroupingPolicy\":\"CombineCompatibleSchemas\"}}"

Upvotes: 3

pdanchenko
pdanchenko

Reputation: 212

Maybe it will be helpful. As per AWS documentation:

Set the Configuration field with a string representation of the following JSON object in the crawler API:

{
   "Version": 1.0,
   "Grouping": {
      "TableGroupingPolicy": "CombineCompatibleSchemas" }
}

Upvotes: -1

Related Questions