CODEWITHSUNDEEP
apacheapache-nifihortonworks-data-platformhortonworks-dataflowapache-minifi
Rishab Prasad
Rishab Prasad

Reputation: 951

How to automate DBCPConnectionPool Controller service in Apache NiFi?

I am new to Apache NiFi and my basic task to ingest data from multiple database sources into HDFS using Apache Nifi. I use 'DBCPConnectionPool' Controller service for database connection pooling service, where I need to provide information such as 'Database Connection URL', 'Database Driver Class Name', 'Database Driver Location', 'Database User', 'Passoword' etc. Now, since I am ingesting from multiple data sources, so everytime the source database changes I manually need to change all the above mentioned information. I am looking for a way so that we can automate this task. For example, The 'DBCPConnectionPool' controller service should be able to dynamically get the 'Database Connection URL','Database Driver Class Name' and other informations depending upon the source database.

Upvotes: 2

Views: 3882

Answers (2)

Christophe De Loeul
Christophe De Loeul

Reputation: 184

You should create an DBCPConnectionPoolLookup controller,

From Nifi's documentation:

Provides a DBCPService that can be used to dynamically select another DBCPService. This service requires an attribute named 'database.name' to be passed in when asking for a connection, and will throw an exception if the attribute is missing. The value of 'database.name' will be used to select the DBCPService that has been registered with that name. This will allow multiple DBCPServices to be defined and registered, and then selected dynamically at runtime by tagging flow files with the appropriate 'database.name' attribute.

The idea is to have a DBCPConnectionPool for each of the database and reference them using the DBCPConnectionPoolLookup.

So, once it's setup you only have to change the database.name attribute from your flowfile i.e. by using an UpdateAttribute

REST is not the way to go here, unless specific actions are needs, because you'll have to keep track of the processors'id's and you'll have to manage the restart of your DBCPConnectionPool which leads to unnecessary development of your REST client/script.

Upvotes: 0

Mahendran V M
Mahendran V M

Reputation: 3496

@Rishab Prasad,

NiFi supports RestAPI for dynamically change the value 'DBCPConnectionPool' properties.

Check below reference but for this you need to create empty cotroller service in an required processors.

https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

If you created empty controller service then that having seperate id. with help of above rest api for update the controller services you can dynamically change its values.

This is only way to dynamically change the DBCP Connection pool.

Otherwise you need to change the nifi source to achieve your requirment.

Upvotes: 2

Related Questions