Elad Project
Elad Project

Reputation: 11

How to get the outputs and artifacts of a pipeline in kubeflow?

I'm using the pythagorean example in kubeflow v2:

@dsl.component
def square(x: float) -> float:
    return x ** 2

@dsl.component(packages_to_install=['numpy==1.21.6', 'pyyaml==6.0'])
def add(x: float, y: float) -> float:
    return x + y

@dsl.component
def square_root(x: float) -> float:
    return x ** .5

@dsl.pipeline
def pythagorean(a: float, b: float) -> float:
    a_sq_task = square(x=a)
    b_sq_task = square(x=b)
    sum_task = add(x=a_sq_task.output, y=b_sq_task.output)
    return square_root(x=sum_task.output).output

I run it on a local one-node k8s cluster (kind) like so:

client = Client(host=host)
run = client.create_run_from_pipeline_func(
    kfp_pipline,
    arguments=kwargs
)
res = run.wait_for_run_completion(timeout=3600)

The pipeline is running and I can see correct results in the UI. How do I access the pipeline's return value in code? If I were to run the pipeline in local DockerRunner like so: res = pythagorean(a=3, b=4) Then res.output will contain the pipeline's return value. But this isn't true when I run on k8s.

If the pipeline were to generate output artifacts, how can I get the list of artifacts it creates (so I can process them)?

I tried looking at the returned data from wait_for_run_completion but there doesn't seem to be any useful info there. I also looked at the REST API, but couldn't find a way to do it (I actually did see an API for reading a specific artifact, but not a way to list them).

Upvotes: 1

Views: 50

Answers (0)

Related Questions