CODEWITHSUNDEEP
pythonworkflowflyte
duncan
duncan

Reputation: 59

Flyte won't run in parallel

I'm new to flyte, trying to do quite simple things at the moment.

So I'm trying to to run multiple independant tasks in parallel, which, if I understand the documentation, is exactly the purpose of map_task.

However, so far I've been unable to make flyte actually run them in parallel.

Any help would be greatly appreciated.

Thanks in advance.

from flytekit import map_task, task, workflow

@task
def do_something(value: str) -> str:
    print(f"launched: {value}", flush=True)
    time.sleep(60)  # fakes long process time
    return f"{value}-processed"


@workflow
def do_multiple_things() -> list[str]:
    values = ["foo", "bar", "baz"]
    return map_task(do_something)(value=values)

Upvotes: 2

Views: 391

Answers (1)

kaylindris
kaylindris

Reputation: 88

Could you give more context please? How are you trying to run it? Is this running in remote (like on a live Flyte backend) or is this a local run? Local runs will not actually do parallel just yet. (Making flytekit execute local runs in parallel is part of a broader project that we have plans for some day, but no definite timeline).

A side note however, as of this writing (circa early 2024) there is a newer version of map task that we are steadily moving towards. If you could, I would suggest using this instead.

from flytekit.experimental import map_task

At some point the main map task will become this one (though we will keep the experimental import in place for compatibility).

Upvotes: 2

Related Questions