CODEWITHSUNDEEP
pythonapache-flinkpyflink
korolkevichm
korolkevichm

Reputation: 45

import local packages inside PyFlink

I'm trying to write a local package in PyFlink project. But I can only import via relative path.

like

from .package import func

Can I use absolute paths in packages inside PyFlink project imported as env.add_python_file('/path_to_project') ?

Upvotes: 0

Views: 729

Answers (1)

korolkevichm
korolkevichm

Reputation: 45

For using absolute paths answer from https://lists.apache.org/[email protected]: full answer here for abstract structure the directory:

flink_app/
    data_service/
        filesystem.py
    validator/
        validator.py
    common/
        constants.py
    main.py <- entry job

When submitting the PyFlink job you can specify python files and entry main module with option --pyFiles and --pyModule1, like:

$ ./bin/flink run --pyModule flink_app.main --pyFiles ${WORKSPACE}/flink_app

In this way, all files under the directory will be added to the PYTHONPAHT of both the local client and the remote python UDF worker.

Upvotes: 2

Related Questions