CODEWITHSUNDEEP
javaloggingapache-flinkapache-storm
cYrus
cYrus

Reputation: 2616

Set Apache Storm and Flink log level to display debug messages

So I'm building a JAR with Storm and Flink applications where I log messages as the following:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

// ...

private static final Logger LOG = LoggerFactory.getLogger(Some.class);

// ...

LOG.debug("...");
LOG.info("...");
LOG.error("...");

Then I pass the JAR to .../bin/storm and .../bin/flink scripts and everything works, but the log level is set to INFO, I'd like to also display DEBUG messages from my package only.

I tried several things but I feel I'm just trying random things from the internet as I can't find an authoritative reference about how to obtain this and I'm having hard time wrapping my head around the incredibly confusing state of the log facilities for Java...

I'm asking about both Storm and Flink as I suspect that the root of my problem is the same but I might be wrong. Also I apologize if I don't provide a minimal example but there's really nothing to provide here.

Please let me know if you need additional details.

Upvotes: 0

Views: 3534

Answers (3)

cYrus
cYrus

Reputation: 2616

In this scenario:

Then I pass the JAR to .../bin/storm and .../bin/flink scripts and everything works, but the log level is set to INFO, I'd like to also display DEBUG messages from my package only.

I ended up with the following suboptimal solution.

Storm

For unknown reasons changing the /path/to/storm/log4j2/worker.xml file has no effect so I need to act programmatically:

import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.core.config.Configurator;

// ...

Configurator.setLevel("my.package", Level.ALL);

Flink

It's enough to add a line to /path/to/flink/conf/log4j.properties:

log4j.logger.my.package=ALL

Upvotes: 0

Till Rohrmann
Till Rohrmann

Reputation: 13346

In order to modify the log level and which classes log on a Flink cluster, please adapt

  • FLINK_HOME/conf/log4j.properties if you are using log4j
  • FLINK_HOME/conf/logback.xml if you are using logback

before you start the Flink cluster.

These files will be read when you deploy the Flink cluster. Note that these settings cannot be changed at runtime, unless you are replacing Flink's log4j logger with log4j2 which supports to load settings dynamically.

Upvotes: 0

Stig Rohde Døssing
Stig Rohde Døssing

Reputation: 3651

For Storm, your log configuration is in storm/log4j2/worker.xml. It's a log4j2 configuration file, so you can find out what options there are by looking at the log4j2 documentation here https://logging.apache.org/log4j/2.x/manual/configuration.html.

I'm not as familiar with Flink, but I'd suspect it's similar. Here is Flink's page on it, which mentions that you should have a logback.xml file in your conf directory.

Upvotes: 0

Related Questions