CODEWITHSUNDEEP
amazon-s3apache-nifi
Archi
Archi

Reputation: 25

read parquet data from s3 bucket using NiFi

guys!

I'm just starting to learn NiFi. Don't throw stones) just help or guide. I need to read parquet data from s3 bucket, I don’t understand how to set up lists3 and fetchs3object processors for reading data. full path looks like this: s3://inbox/prod/export/date=2022-01-07/user=100/ 2022-01-09 06:51:23 23322557 cro.parquet I"ll write data to sql database - I don"t have problems with it.

I tried to configure the lists3 processor myself and I think is not very good bucket inbox aws_access_key_id aws_secret_access_key region US EAST endpoint override URL http://s3.wi-fi.ru:8080

Upvotes: 1

Views: 583

Answers (1)

Mike R
Mike R

Reputation: 588

What I would do is try to test the Access Key ID, and Secret Key outside of NiFi to make sure that they are working. If they are working fine, then it’s an issue with the NiFi configuration. If the keys/id isn’t working, then by getting new values that work and providing them to NiFi, it might have a better shot of working.

Upvotes: 1

Related Questions