CODEWITHSUNDEEP
apache-nifikylo
sanket Manwatkar
sanket Manwatkar

Reputation: 11

Kylo Template failing

I have created the Nifi workflow having 3 processor.Attached are the details. The problem is when I am running the workflow in Nifi it is running fine but when I am importing the same template in Kylo and running it through feed, it is giving error.It seems there is something wrong with Kylo template. Can you please help me with that. Also in the Nifi logs I am not able to see any error.

Thanks,

Kylo Job Failure Kylo Job Failure

Nifi Workflow Nifi Workflow

Kylo Template-1 Kylo Template-1

Kylo Template -2

    Nifi log:

    2017-08-30 10:11:49,764 INFO [pool-8-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@65b65bdb checkpointed with 1 Records and 0 Swap Files in 34 milliseconds (Stop-the-world time = 16 milliseconds, Clear Edit Logs time = 15 millis), max Transaction ID 37082
    2017-08-30 10:11:49,764 INFO [pool-8-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 1 records in 34 milliseconds
    2017-08-30 10:13:47,109 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2efe3a02 checkpointed with 2326 Records and 0 Swap Files in 18 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID 6977
    2017-08-30 10:13:49,764 INFO [pool-8-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
    2017-08-30 10:13:49,799 INFO [pool-8-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@65b65bdb checkpointed with 1 Records and 0 Swap Files in 34 milliseconds (Stop-the-world time = 16 milliseconds, Clear Edit Logs time = 15 millis), max Transaction ID 37082
    2017-08-30 10:13:49,799 INFO [pool-8-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 1 records in 34 milliseconds
    2017-08-30 10:14:05,620 INFO [StandardProcessScheduler Thread-3] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled ExecuteSQL[id=ef6f1e38-682f-4300-d33f-ceb1839cdf61] to run with 1 threads
    2017-08-30 10:14:06,147 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@2678d4a7 // Another save pending = false
    2017-08-30 10:14:08,057 INFO [FeedStatisticsManager-SendStats-2] c.t.n.p.j.ProvenanceEventActiveMqWriter SENDING Batch Events to JMS ProvenanceEventRecordDTOHolder{events=3} 
    2017-08-30 10:14:14,951 INFO [Provenance Maintenance Thread-1] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 46037
    2017-08-30 10:14:14,965 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (4 records) into single Provenance Log File /opt/data/provenance_repository/46033.prov in 18 milliseconds
    2017-08-30 10:14:14,965 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 3 records
    2017-08-30 10:14:38,057 INFO [FeedStatisticsManager-SendStats-0] c.t.n.p.j.ProvenanceEventActiveMqWriter SENDING Batch Events to JMS ProvenanceEventRecordDTOHolder{events=3} 
    2017-08-30 10:14:44,959 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 46041
    2017-08-30 10:14:44,999 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (4 records) into single Provenance Log File /opt/data/provenance_repository/46037.prov in 39 milliseconds
    2017-08-30 10:14:44,999 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 3 records

Upvotes: 0

Views: 396

Answers (1)

Greg Hart
Greg Hart

Reputation: 386

Kylo determines if a flow file has failed by looking at the connection name. Since you're using the same connection for failure and success then Kylo is unable to determine whether it is a failure or success.

There's a few ways to change this:

  1. Add a PutFile or LogAttribute processor for failures and delete your existing failure connections. This will stop the flow file from progressing if there's a failure.

  2. Auto-terminate the failure connection and delete your existing failure connections. This will stop the flow file from progressing but Kylo will display everything as a success since there is no failure connections.

  3. Rename your connections to "success". This will allow the flow file to continue and Kylo will display everything as a success.

Upvotes: 5

Related Questions