CODEWITHSUNDEEP
apache-nifi
Jeeva Bharathi
Jeeva Bharathi

Reputation: 554

Why does QueryDatabaseTable doing a complete query fetch instead of using maximum column value to fetch data from Oracle in Apache Nifi?

I am using QueryDatabaseTable processor to do an incremental batch update to Bigquery. Oracle Database Table keeps on increasing at the rate of 5 new rows per minute.

Flow: QueryDatabaseTable -> ConvertAvroToJson -> PutBigQueryBatchUpdate

I ran this flow with a schedule of 10 minute, the query results in about 2000 rows.

QueryDatabaseTable processor configuration I have modified:

Table Name, Additional WHERE clause, Maximum-value Columns.

QueryDatabaseTable is supposed only fetch after the maximum value of the column visible in 'View State'. But my setup simply return the entire result for query.

After each query the maximum value of the column is updated to the latest maximum value.

The maximum value of the column contains a Date.

I have also tried running after clearing the state, and with no values Maximum-value Columns empty, same result.

What am I missing?

Additional info: QueryDatabaseTable config also has this following section, which I think is related to this issue,

Transaction Isolation Level : No value set

Upvotes: 1

Views: 811

Answers (1)

Jeeva Bharathi
Jeeva Bharathi

Reputation: 554

QueryDatebaseTable did not work if I gave just the table name.

Removing the WHERE clause property and creating Custom query made the processor work as intended.

Upvotes: 1

Related Questions