CODEWITHSUNDEEP
palantir-foundry
Arvind
Arvind

Reputation: 301

Streaming data in palantir foundry

I have implemented the streaming pipeline using pipeline builder and everything works fine. It get updated every 5 secs. Now I have to implement the same logic using code repository. Wanted to check if anyone have already tried to implement the same and how does it work? Is there any specific decorator or just @increment should be used?

Upvotes: 0

Views: 90

Answers (1)

ZettaP
ZettaP

Reputation: 1379

Pipeline Builder is the main way to build streaming pipelines.

To my knowledge, you can't use a Code Repository to define your Streaming pipelines the same way you do with Pipeline Builder.

However, you can use Code Repositories to define Streaming UDFs, which then can be called from Pipeline builder like any other transformation. You can essentially build your own "library" of function (which can output 1 or N rows, keyed or not, etc.) and use those bricks in Pipeline Builder. https://www.palantir.com/docs/foundry/transforms-java/user-defined-functions#publish-to-a-deployment-repository

Upvotes: 0

Related Questions