CODEWITHSUNDEEP
dockerapache-nifi
user3472
user3472

Reputation: 313

How to create a running nifi docker image with a flow

I launched a container of apache/nifi, built, and configured a flow.

I'd like to somehow save that flow off somewhere, so that it can be loaded into a new docker image running Nifi.

Such that, a 'user' only has to do 'docker run....' and an instance of nifi will be launched, the flow loaded, and started.

It's not clear to me what files (nar, xml, etc...) need to be made available to the image a user is to run.

Upvotes: 1

Views: 2591

Answers (2)

pirods
pirods

Reputation: 43

You could use the nifi-toolkit to deploy flows and process groups to your Apache NiFi instance without having to rely on the GUI.

https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html

This setup required you having:

  • Apache NiFi
  • Apache NiFi-Registry

Here is a working example (provided that the hostname of your Apache NiFi-Registry container is nifi-registry and its port the default 18080) based on an empty Apache NiFi instance and Nifi-Registry. Tested on Apache NiFi 1.12.1.

Firstly you need to generate a JSON file for your flow through the Apache Registry.

  1. Add a Registry to your Apache NiFi:

/opt/nifi/nifi-toolkit-current/bin/cli.sh nifi create-reg-client -rcn registry -rcu http://nifi-registry:18080

Create a Process Group that will contain your flow. Right click on it and click on "Version" and "Start Version Control". This will save your flow inside the NiFi Registry. Work on your flow through the GUI and when you are ready, right click on your process group and commit your last changes. Now you will need to export the JSON of your flow from the registry.

/opt/nifi/nifi-toolkit-current/bin/cli.sh registry export-flow-version -u http://nifi-registry:18080 -f <flowid> -fv <flowversion> > <json_file>

Now that your JSON flow is ready, you are ready to deploy it on a fresh environment.

  1. Create a bucket inside the registry. This will return the newly generated bucket id.

/opt/nifi/nifi-toolkit-current/bin/cli.sh registry create-bucket -u http://nifi-registry:18080 -bn <bucketname>

  1. Use the previously generated bucket id to create a flow. This will return the newly generated flow id:

/opt/nifi/nifi-toolkit-current/bin/cli.sh registry create-flow -u http://nifi-registry:18080 -b <bucketid> -fn <flowname>

  1. Import your flow (it must have been previously export from the GUI -> right click download flow, and available in the Apache NiFi filesystem):

/opt/nifi/nifi-toolkit-current/bin/cli.sh registry import-flow-version -u http://nifi-registry:18080 -f <flowid> -i <json_file>

  1. Deploy the flow as a process group. This will return the newly generated process group id.

/opt/nifi/nifi-toolkit-current/bin/cli.sh nifi pg-import -b <bucketid> -f <flowid> -fv <flowversion>

  1. Start the process group services (if any)

/opt/nifi/nifi-toolkit-current/bin/cli.sh nifi pg-enable-services -pgid <processgroupid>

  1. Start the processors of your process group (if any):

/opt/nifi/nifi-toolkit-current/bin/cli.sh nifi pg-start -pgid <processgroupid>

Please keep in mind that Apache NiFi should be up and running before executing these commands. If you are planning on embedding these instructions in the Dockerfile, some logic that waits for the service to be up should be implemented.

You might also take a look at this Python wrapper for the NiFi Toolkit:

https://github.com/Chaffelson/nipyapi

Lastly, Apache NiFi also provides some REST APIs that might help you:

https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

Upvotes: 2

maxime G
maxime G

Reputation: 1771

if you have nothing custom you can save the flow.xml.gz from the /conf directory to save the flow.

if you also want to save the content flowfile or current flowfile, you should also save the flowfile repository and content repository

if you have customs processors you should save the nar in lib directory.

everything should be present in the nifi directory before starting it.

Upvotes: 2

Related Questions