CODEWITHSUNDEEP
pythonkubeflowkubeflow-pipelines
mabrtn
mabrtn

Reputation: 21

Python OOP in Kubeflow Pipelines

I have a complete preprocessing, training, evaluating ML pipeline in Python 3.8 that I want to transform in a Kubeflow pipeline. It's mainly for segmentation task (i.e. on RGB images)

All my code is design in a OOP way. Is this possible to keep the OOP strategy in Kubeflow ? By this question I mean :

If it's possible, if you have any code samples as exemples it will be helpful.

Otherwise, what do you recommend to move to KF easily ?
Thanks for your help ;)

Upvotes: 2

Views: 502

Answers (1)

Hamza Tahir
Hamza Tahir

Reputation: 446

I don't know if there is a native Kubeflow way of doing this, but as func_to_container_op takes a standalone function as its input, the self state variable will surely be a problem. You might have some luck by inspecting the source code of the func_to_container_op function, which eventually uses a factory to create a class which you might create directly.

As a simpler alternative, if you look at the class-based API of ZenML, you can do something like this:


class Preprocessing(BaseStep):
    def entrypoint(self,
                   dataset: pd.DataFrame,
                   ) -> Output(train=pd.DataFrame, 
                               test=pd.DataFrame, 
                               validation=pd.DataFrame):
        
        ...
        return train_dataset, test_dataset, val_dataset

This step can then be put a in a Pipeline and deployed with Kubeflow as the orchestrator [see here]

As the step is already a class, perhaps there is an easy way for you to convert your existing classes to ZenML BaseStep classes?

Disclaimer: I'm one of the co-founders of ZenML.

Upvotes: 1

Related Questions