CODEWITHSUNDEEP
google-cloud-dataflowgoogle-cloud-ml
jjreddick
jjreddick

Reputation: 285

Google cloud dataflow and machine learning

What's the best way to run machine learning algorithms on Google Cloud Dataflow? I can imagine that using Mahout would be one option given it's Java based.

The answer is probably no, but is there a way to invoke R or Python (that have strong support for algorithms) based scripts to offload ML execution?

-Girish

Upvotes: 2

Views: 912

Answers (2)

Davor Bonaci
Davor Bonaci

Reputation: 1729

You can already implement many algorithms in terms of Dataflow transforms.

A class of algorithms that may not be as easy to implement are iterative algorithms, where the pipeline's execution graph depends on the data itself. Simplifying implementation of iterative algorithms is something that we are interested in, and you can expect future improvements and simplifications in this area.

Invoking Python (or any other) executable shouldn't be hard from a Dataflow pipeline. A ParDo can, for example, shell out and start an arbitrary process. You can use, for example, --filesToStage pipeline option to add additional files to the Dataflow worker environment.

Upvotes: 1

G B
G B

Reputation: 755

There is also http://quickml.org/ (haven't used personally) and Weka. I remember the docs mention that it's possible to launch a new process from within the job, but AFAIK it's not recommended.

Upvotes: 0

Related Questions