CODEWITHSUNDEEP
databaseetlapache-nifi
bp2010
bp2010

Reputation: 2472

NiFi - QueryDatabaseTable for 100s of tables - Generic Solution

Currently have a flow using QueryDatabaseTable which reads from a DB and puts the data into HDFS. Decided to use QueryDatabaseTable because:

My question is that I now have 100 tables that require the same flow (DB => HDFS). I do not want to create the same flow 100 times. I have looked into ListDatabaseTables which would be perfect, but it seems QueryDatabaseTable doesn't take any input.

Has anyone encountered something similar?

Upvotes: 1

Views: 471

Answers (1)

Bryan Bende
Bryan Bende

Reputation: 18640

QueryDatabaseTable is meant to do incremental loading of a table and therefore has to maintain state about the table so it can now what to retrieve on next execution. As a result, it can't allow dynamic tables because then there is an infinite amount of state that needs to be kept.

ListDatabaseTables is meant to be used more with GenerateTableFetch and ExecuteSQL to do bulk loading of a DB table.

Upvotes: 2

Related Questions