CODEWITHSUNDEEP
google-cloud-dataflowapache-beam
janneh
janneh

Reputation: 245

How to set log level for apache beam pipeline running on Direct Runner

When running an Apache Beam pipeline locally using Direct Runner the log level seems to be set to DEBUG.

Is there a way to set the log to say INFO instead?

Note: --workerLogLevelOverrides can be used when using Cloud Dataflow Runner, but does not seem to apply on Direct Runner

Upvotes: 5

Views: 7584

Answers (1)

Adrian
Adrian

Reputation: 1595

It appears that per standard configuration, the logging is done with slf4j using a JUL(java.util.logging) backend. You can check in your pom.xml for slf4j-api which corresponds to slf4j and slf4j-jdk14 for its backend.

You need to configure JUL at runtime. One way is to pass as a parameter to the VM something like

-Djava.util.logging.config.file=logging.properties

Then you create a logging.properties in the directory where you run your application. In my case it is the root directory for the app(the one that contains src directory). If it doesn't seem to work, you can just specify an absolute path in as the filename.

Regarding the configuration, here you have an example:

handlers=java.util.logging.ConsoleHandler
.level=INFO
java.util.logging.ConsoleHandler.level=ALL
org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource.level=ALL

This sets the level for the package org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource. Depending on the situation you can play with it's configuration. Make sure that the level is properly set to all the components involved in JUL log output.

Upvotes: 5

Related Questions