CODEWITHSUNDEEP
apache-flink
IsabelleR
IsabelleR

Reputation: 3

How can I access states computed from an external Flink job ? (without knowing its id)

I'm new to Flink and I'm currently testing the framework for a usecase consisting in enriching transactions coming from Kafka with a lot of historical features (e.g number of past transactions between same source and same target), then score this transaction with a machine learning model.

For now, features are all kept in Flink states and the same job is scoring the enriched transaction. But I'd like to separate the features computation job from the scoring job and I'm not sure how to do this.

  1. The queryable state doesn't seem to fit for this, as the job id is needed, but tell me if I'm wrong !

  2. I've thought about querying directly RocksDB but maybe there's a more simple way ?

  3. Is the separation in two jobs for this task a bad idea with Flink ? We do it this way for the same test with Kafka Streams, in order to avoid complex jobs (and to check if it has any positive impact on latency)

Some extra information : I'm using Flink 1.3 (but willing to upgrade if it's needed) and the code is written in Scala

Thanks in advance for your help !

Upvotes: 0

Views: 539

Answers (1)

David Anderson
David Anderson

Reputation: 43499

Something like Kafka works well for this kind of decoupling. In that way you could have one job that computes the features and streams them out to a Kafka topic that is consumed by the job doing the scoring. (Aside: this would make it easy to do things like run several different models and compare their results.)

Another approach that is sometimes used is to call out to an external API to do the scoring. Async I/O could be helpful here. At least a couple of groups are using stream SQL to compute features, and wrapping external model scoring services as UDFs.

And if you do want to use queryable state, you could use Flink's REST api to determine the job id.

There have been several talks at Flink Forward conferences about using machine learning models with Flink. One example: Fast Data at ING – Building a Streaming Data Platform with Flink and Kafka.

There's an ongoing community effort to make all this easier. See FLIP-23 - Model Serving for details.

Upvotes: 3

Related Questions