CODEWITHSUNDEEP
apache-nifi
Adelson Araújo
Adelson Araújo

Reputation: 342

Looping an API using NiFi with different URLs

I'm a freshman using NiFi. I have the following NiFi dataflow getting data from GET request on a fixed URL, but my endpoint has a set of parameters to fetch parts of the data, for example the page id. I want to avoid making this same workflow for a slightly different request URL.

In the API, I can request for the list of pages available, and I want to use the result of such request to start looping the workflow below. How would I do this??

enter image description here

Upvotes: 0

Views: 578

Answers (1)

Mike Thomsen
Mike Thomsen

Reputation: 37506

Probably your easiest way to do this would be to add expression language statements to parameterize the URL in InvokeHTTP and then put ExecuteScript (Cookbook) above it with a Groovy or JRuby (don't use Jython!) script to generate flowfiles that have attributes that correspond to the parameter names.

This is a simple script that would generate a request for each page for a url with a parameter called tag:

def pages = 10

1.upto(pages) { page ->
  def ff = session.create();
  ff = session.putAttribute(ff, "tag", "articles")
  ff = session.putAttribute(ff, "page", page.toString())
  session.transfer(ff, REL_SUCCESS)
}

The attributes are then visible with expression language as ${tag} and ${page}. See documentation for more complex scripting examples. You'll want to make sure that InvokeHTTP is set to write the REST response to the body of the flowfile (that's a configuration option).

Upvotes: 2

Related Questions