CODEWITHSUNDEEP
apache-nifisqlconnection
Tyr
Tyr

Reputation: 610

How to read sql table on Nifi?

I am trying to create a basic flow on Nifi

  1. read table from sql
  2. process it on python
  3. write back another table in sql

It is simple as it is.

Nifi flow

But, I am facing issues when I try to read data on python

As far as I learn I need to use sys.stdin/out. It only reads and writes as below.

import sys
import pandas as pd

file = pd.read_csv(sys.stdin)
file.to_csv(sys.stdout,index=False)

Below you can find processor properties, but I don't think it is the issue.

QueryDatabaseTableRecord:

QueryDatabaseTableRecord

ExecuteStreamCommand: enter image description here

PutDatabaseRecord: enter image description here

Error Message:

enter image description here

Upvotes: 0

Views: 913

Answers (1)

Mike Thomsen
Mike Thomsen

Reputation: 37506

There's a much easier way to do this if you're running 1.12.0 or newer: ScriptedTransformRecord. It's like ExecuteScript except it works on a per-record basis. This is what a simple Groovy script for it looks like:

def fullName = record.getValue("FullName")
def nameParts = fullName.split(/[\s]{1,}/)
record.setValue("FirstName", nameParts[0])
record.setValue("LastName:", nameParts[1])
record

It's a new processor, so there's not that much documentation on it yet aside from the (very good) documentation bundled with it. So samples might be sparse at the moment. If you want to use and run into issues, feel free to join the nifi-users mailing list and asked for more detailed help.

Upvotes: 1

Related Questions