NIFI - Dev to Test to Prod

Question

We are struggling with trying to figure out the best approach for updating processor configurations as a flow progresses through the dev, test, and prod stages. We would really like to avoid manipulating host, port, etc. references in the processors when the flow is deployed to the specific environment. At least in our case, we will have different hosts for things like ElasticSearch, PostGres, etc. How have others handled this?

Things we have considered:

• Pull the config from a properties file using expression language. This is great for processors that have EL enabled, but not the case for those where it isn't.
• Manipulate the flow xml and overwrite the host, port, etc. configurations. A bit concerned about inadvertently corrupting the xml and how portable this will be across NIFI versions.

Any tips or suggestions would be greatly appreciated. There is a good chance that there is an obvious solution we have neglected to consider.

EDIT:

We are going with the templates that Byran suggested. They will definitely meet our needs and appear to be a good way for us to control configurations across numerous environments.

https://github.com/aperepel/nifi-api-deploy

Answer 1

You can loook at this post automating NIFI template deployment

For automating NIFI template deployment, there is a tool that works well : https://github.com/hermannpencole/nifi-config

• Prepare your nifi development

  1. Create a template on nifi
  2. and download it
  3. Extrac a sample configuration with the tools
• Deploy it on production
  1. undeploy the old version with the tools
  2. deploy the template with the tools
  3. update the production configuration with the tools

Answer 2

This discussion comes up frequently, and there is definitely room for improvement here...

You are correct that currently one approach is to extract environment related property values into the bootstrap.conf, and then reference them through expression language so the flow.xml.gz can be moved from one environment to the other. As you mentioned this only works well with properties that support expression language.

In order to make this easier in the future, there is a feature proposal for an idea called a Variable Registry:

https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry

An interesting approach you may want to look at is using templates. There is a GitHub project that can be used to help with this:

https://github.com/aperepel/nifi-api-deploy