CODEWITHSUNDEEP
apache-nifi
Japson Zhang
Japson Zhang

Reputation: 13

how to get list of flowfile from queue in nifi connection?

I want to get each flowfile from queue in connection when the flow was blocked with NiFi REST API.

I find that the functions of /nifi-api/flowfile-queues/{id}/listing-requests can meet my needs.And it response a ListingRequestDTO: enter image description here

The field named flowFileSummaries can return a array of flowFile.And then I can get each uuid from flowfiles:

{
"uri": "value",
"uuid": "value",
"filename": "value",
"position": 0,
"size": 0,
"queuedDuration": 0,
"lineageDuration": 0,
"clusterNodeId": "value",
"clusterNodeAddress": "value",
"penalized": true

}

BUT,when I use the API and can not find the "FlowFileSummary"

{
"listingRequest": {
    "id": "0165122a-e1ac-134e-2c09-92ba9ca93e8b",
    "uri": "http://.../nifi-api/flowfile-queues/07a23828-d6f3-1e00-27af-f0428a493507/listing-requests/0165122a-e1ac-134e-2c09-92ba9ca93e8b",
    "submissionTime": "09/07/2018 18:46:57.496 CST",
    "lastUpdated": "18:46:57 CST",
    "percentCompleted": 0,
    "finished": false,
    "maxResults": 100,
    "state": "Waiting for other queue requests to complete",
    "queueSize": {
        "byteCount": 370689,
        "objectCount": 995
    },
    "sourceRunning": false,
    "destinationRunning": false
}

}

So, is there any possible solution to achieve? or any other solution? Thanks!

Upvotes: 1

Views: 3944

Answers (2)

Ramprasad
Ramprasad

Reputation: 1

You do not need to have the listing request to complete. Here are the steps I used successfully to get the contents of flow file

  1. Get the id of the Queue from NiFi UI console
  2. Replace id here : curl -X POST https://myhost:443/nifi-api/flowfile-queues/{id}/listing-requests -H 'Authorization: Bearer ' --compressed --insecure
  3. To get uris of flowfiles in the queue, get the value of 'uri' field from the response of the command in step2 and use it in next curl command, for ex: For ex: curl -X GET https://myhost:443/nifi-api/flowfile-queues/0f66c88c-225d-3229-b2e1-597d8fba2c09/listing-requests/13802f9c-016a-1000-0000-00004eb033fb -H 'Authorization: Bearer ' --compressed --insecure
  4. To get flow file content, get the value of 'uri' field from the response of the command in step3 and and use it in next curl command, for ex: curl -X GET https://myhost:443/nifi-api/flowfile-queues/0f66c88c-225d-3229-b2e1-597d8fba2c09/flowfiles/7ccf5c54-7c8d-448a-a124-7f75f5845ec1?clusterNodeId=35a3df65-d7bf-47d2-b162-ea15c3c02c30 -H 'Authorization: ' --compressed --insecure

Note: if there are multiple flow files, you will see see multiple URIs in the response of step 3 curl command

Upvotes: 0

mattyb
mattyb

Reputation: 12093

I don't believe the flow file summaries are available until the listing request is complete. In your example response above, note that finished is false. You can keep querying the API for that listing request until finished is true, then the summaries should be available. Then you can use the /flowfile-queues/{id}/flowfiles/{flowfile-uuid} and /flowfile-queues/{id}/flowfiles/{flowfile-uuid}/content endpoints to get the attributes and content of each flow file.

Upvotes: 3

Related Questions