CODEWITHSUNDEEP
apache-hop
mono1234
mono1234

Reputation: 21

Apache Hop Kafka Consumer throws error when I run pipeline using the Flink Engine

I am trying to test a simple kafka consumer using Apache Hop v1.2. When I run the pipeline using the local runner, it works fine. But if I run it using the flink runner I get the following error

You can only have one copy of the injector transform 'output' to accept the Kafka messages

I have tried debugging the Hop code and looks like the root cause is the initSubPipeline() method being invoked multiple times while using the Flink runner. That's not the case when I use the local runner. Am I missing something here?

Upvotes: 0

Views: 233

Answers (1)

HansVA
HansVA

Reputation: 106

For others bumping into this problem, I discussed it on an other communication platform with Mono.

In this case the problem lies with the Transform. Not all transforms in Hop are able to run in a clustered environment such as Flink, for Apache Kafka we have 2 transforms. One is called "Beam Kafka Consumer/Producer" this one can be used in combination of one of our engines that use Apache Beam (Flink, Spark, Google Dataflow). The other are the "regular" Kafka producer/consumer and these will only work using the Hop Engine.

There is a ticket on our backlog (HOP-3863) that will add a system to throw an error or warning when using a transform that is not made to run on one of our engines.

Upvotes: 1

Related Questions