CODEWITHSUNDEEP
jsonhortonworks-data-platformapache-nifi
Shikhar Agarwal
Shikhar Agarwal

Reputation: 56

How can I use Apache NiFi to retrieve attributes after splitting a JSON and use those attributes recursively in the invokeHttp processor?

I'm accessing the weather data from an api and want to use Apache nifi to get the weather data for all the cities in the json array returned by the first api?

Basically I'm accessing the data through 2 APIs:

This api returns me a JSON array containing the list of all cities in USA in a JSON array element:

{
  "response": {
  "version":"0.1",
  "termsofService":"http://www.wunderground.com/weather/api/d/terms.html",
  "features": {
  "conditions": 1
  }
        , "results": [
        {
        "name": "Keyhole",
        "city": "Keyhole",
        "state": "WY",
        "country": "US",
        "country_iso3166":"US",
        "country_name":"USA",
        "zmw": "82721.7.99999",
        "l": "/q/zmw:82721.7.99999"
        }
        ,
        {
        "name": "Cuchara Valley Airport At La Veta",
        "city": "Cuchara Valley Airport At La Veta",
        "state": "CO",
        "country": "US",
        "country_iso3166":"US",
        "country_name":"USA",
        "zmw": "81055.6.99999",
        "l": "/q/zmw:81055.6.99999"
        }
        ,

Now I want to iterate through this array and use the state and city information to be used in another API like below:

{
  "response": {
  "version":"0.1",
  "termsofService":"http://www.wunderground.com/weather/api/d/terms.html",
  "features": {
  "conditions": 1
  }
    }
  , "current_observation": {
        "image": {
        "url":"http://icons.wxug.com/graphics/wu2/logo_130x80.png",
        "title":"Weather Underground",
        "link":"http://www.wunderground.com"
        },
        "display_location": {
        "full":"San Francisco, CA",
        "city":"San Francisco",
        "state":"CA",
        "state_name":"California",
        "country":"US",
        "country_iso3166":"US",
        "zip":"94101",
        "magic":"1",
        "wmo":"99999",
        "latitude":"37.77500916",
        "longitude":"-122.41825867",
        "elevation":"47.00000000"
        },
        "observation_location": {
        "full":"SOMA - Near Van Ness, San Francisco, California",
        "city":"SOMA - Near Van Ness, San Francisco",
        "state":"California",
        "country":"US",
        "country_iso3166":"US",
        "latitude":"37.773285",
        "longitude":"-122.417725",
        "elevation":"49 ft"
        },...

So basically I've to run the second api in a loop for all the combination of state and city which I'm getting from the first API.

The basic NiFi flow which I've made so far is as attached to this question but it's not working. Can someone please help me figure out what's that I'm doing wrong here?

Hortonworks Dataflow on HDP 2.4 Sandbox

SplitJson Processor Configuration

Upvotes: 2

Views: 2108

Answers (1)

JDP10101
JDP10101

Reputation: 1852

Answering the problems exposed in comments:

For the SplitJSON the JSON path expression you want to use is *.results.* this will split the JSON into a FlowFile per result. The EvaluateJSONPath configuration you have should work with an updated SplitJSON (just tested in a template).

The Expression Language (EL) you provided for the InvokeHttp URL is almost there. One thing you will need is to URL encode the city and state using the EL "urlEncode" like so: api.wunderground.com/api/key/conditions/q/${state:urlEncode()}/${city:urlEncode()}.j‌​son

Upvotes: 2

Related Questions