CODEWITHSUNDEEP
pythonloggingstdoutprefect
Oleh Rybalchenko
Oleh Rybalchenko

Reputation: 8059

Forwarding stdout and stderr to logger in Prefect 2 flows

I have a couple of old ETL jobs that need to be adopted to Prefect flows. All of them are utilizing print statements for logging. Jobs should be backward-compatible and keep printing existing messaged to stdout and stderr. The issue is that stout and stderr messages are ignored by Orion UI for viewing logs.

Long story short, I simply need stdout and stderr messages handled as logger.info and logger.warning respectively.

In Prefect v1 there was a native option to forward stdout to logger. However, it's removed in v2.

Anything native I'm missing here?

Upvotes: 2

Views: 1077

Answers (2)

Anna Geller
Anna Geller

Reputation: 1758

Fully agree that it should be equally easy as in v1, and we have an open roadmap item for that: https://github.com/PrefectHQ/prefect/issues/7117

Definitely on our radar and coming soon

Upvotes: 1

Oleh Rybalchenko
Oleh Rybalchenko

Reputation: 8059

For now I've made a workaround that works fine, but seems pretty heavy comparing to how it was in v1.

class Logger:
    def __init__(self, level):
        self.level = level

    def write(self, message):
        if message != '\n':
            self.level(message)

    def flush(self):
        self.level(sys.stderr)


def setup_logger():
    """
    Adopt stdout and stderr to prefect logger
    :return:
    """
    prefect_logger = get_run_logger()
    sys.stdout = Logger(prefect_logger.info)
    sys.stderr = Logger(prefect_logger.error)
    return prefect_logger

@flow
def transform():
    logger = setup_logger()
    print('Hello stdout')
    logger.info('Hello logger')

Which results in the following log:

14:00:10.681 | INFO    | prefect.engine - Created flow run 'pygmy-eagle' for flow 'transform'
14:00:11.075 | INFO    | Flow run 'pygmy-eagle' - Hello stdout
14:00:11.076 | INFO    | Flow run 'pygmy-eagle' - Hello logger
14:00:11.105 | INFO    | Flow run 'pygmy-eagle' - Finished in state Completed()

Upvotes: 3

Related Questions