CODEWITHSUNDEEP
amazon-web-servicesaccess-tokenapache-nifi
Ani
Ani

Reputation: 608

How to use Aws Temporary credentials in Nifi

I have to use aws temporary credentials AccessKey, SecretKey and Token within nifi process to access S3 objects. AccessKey, SecretKey and Token will be provided by an Api call. How to use these temperory credentials in nifi ListS3 Object etc?

One of the options I found is using AWSCredentialsProviderControllerService since this Can I update the values of a controller service based on an API call?

AwsCrentialsProviderControllerService

Any other options that I can use in this scenario?

Thanks! Ani

Upvotes: 3

Views: 2705

Answers (1)

Sivaprasanna Sethuraman
Sivaprasanna Sethuraman

Reputation: 4132

Yeah, we can leverage AWSCredentialsProvider controller service and NiFi REST API to build this although it may be a bit complex. Nevertheless, let's sketch it out here.

Whatever action you do on NiFi - updating the processor/controller service configuration, stopping/starting services/processors, are all making API calls in the backend. NiFi framework makes the call behind the scenes. And since we also have HTTP processors like InvokeHTTP we could build this thing. I could think of a something like the below:

  1. Create AWSCredentialsProvider controller service and leave the configurations empty which effectively renders the component to be in invalid state. Note down its unique ID (UUID). Let's call it cp-id. We'll use it later.
  2. Create your flow with your logic that involves ListS3 -> <OTHER_PROCESSORS>. Configure the ListS3 processor with the AWSCredentialsProvider we created in Step #1.
  3. Note down the component ID of this ListS3 processor. Let's call it s3-id
  4. Create another flow using InvokeHTTP and configure it to your service endpoint which gives you your temporary AWS credentials. Use the credentials in and make another call with InvokeHTTP to NiFi server with PUT /controller-services/{cp-id} endpoint to update the properties
  5. Make another call to update the run status of the controller service with PUT /controller-services/{cp-id}/run-status. This will enable the controller service which means now you can start the ListS3 processor which can again be done again using a HTTP call to NiFi server: PUT /processors/{s3-id}/run-status

Some useful links:

Upvotes: 4

Related Questions