CODEWITHSUNDEEP
amazon-web-servicesaws-glueamazon-athenaamazon-kinesisaws-glue-data-catalog
madej
madej

Reputation: 59

Glue crawler is not combining data - also no visible data in tables

I'm testing this architecture: Kinesis Firehose → S3 → Glue → Athena. For now I'm using dummy data which is generated by Kinesis, each line looks like this: {"ticker_symbol":"NFLX","sector":"TECHNOLOGY","change":-1.17,"price":97.83}

However, there are two problems. First, a Glue Crawler creates a separate table per file. I've read that if the schema is matching Glue should provide only one table. As you can see in the screenshots below, the schema is identical. In Crawler options, I tried ticking Create a single schema for each S3 path on and off, but no change.

Files also sit in the same path, which leads me to the second problem: when those tables are queried, Athena doesn't show any data. That's likely because files share a folder - I've read about it here, point 1, and tested several times. If I remove all but one file from S3 folder and crawl, Athena shows data.

Can I force Kinesis to put each file in a separate folder or Glue to record that data in a single table?

File1: File 1

File2: File 2

Upvotes: 0

Views: 3745

Answers (1)

Amir Maleki
Amir Maleki

Reputation: 419

Regarding the AWS Glue creating separate tables there could be some reasons based on the AWS documentation:

  1. Confirm that these files use the same schema, format, and compression type as the rest of your source data. It seems this doesn't your issue but still to make sure I suggest you test it with smaller files by dropping all the rows except a few of them in each file.
  2. combine compatible schemas when you create the crawler by choosing to Create a single schema for each S3 path. For this case, file schema should be similar, setting should be enabled, and data should be compatible. For more information, see How to Create a Single Schema for Each Amazon S3 Include Path.
  3. When using CSV data, be sure that you're using headers consistently. If some of your files have headers and some don't, the crawler creates multiple tables

One another really important point is, you should have one folder at root and inside it, you should have partition sub-folders. If you have partitions at S3 bucket level, it will not create one table.(mentioned by Sandeep in this Stackoverflow Question)

I hope this could help you to resolve your problem.

Upvotes: 1

Related Questions