CODEWITHSUNDEEP
javaslf4jreact-contextmdcmicrometer-tracing
Rahul verma
Rahul verma

Reputation: 314

How to propagate and clear thread context across JVMs for dynamic debug logging

I am trying to implement dynamic debug logging by populating a ThreadContext with specific attributes and filtering logs based on those attributes.

Here’s what I have achieved so far:

Within the same JVM:

  1. For reactive flows, the thread context is passed correctly to all reactive and non-reactive flows using the contextWrite() method and cleared after the chain is completed.
  2. For ExecutorService, I wrap the Callable in a ContextExecutorService to manage the ThreadContext.
  3. answer referred : Dynamically set DEBUG or INFO logging level per request based on request header in reactive Spring Boot with Log4j2

Issue across JVMs:

In a distributed system (e.g., Hazelcast), where a process is submitted to another JVM, the context must be manually passed in a field and set in the thread context of the target JVM. example process that is submitted to another JVM :

@Slf4j
public class LoggingEntryProcessor<K, V, R> implements EntryProcessor<K, V, R>, HazelcastInstanceAware {

  private final EntryProcessor<K, V, R> processor;
  private Map<String, String> attributes;

  @Override
  public R process(Entry<K, V> entry) {
    // Ensure thread context is populated with attributes received from caller JVM and cleared after work is done
    return processor.process(entry); 
    //the process method is nonreactive and may have reactive code inside of it. 
  }
  
  private ContextSnapshot setContextSnapshot(Map<String, String> attributes) {
    // tried creating a snapshot with which i can wrap, Slf4jThreadLocalAccessor is registered in ContextRegistry in this JVM
    new Slf4jThreadLocalAccessor().setValue(attributes);
    return ContextSnapshotFactory.builder().build().captureAll();
  }
}

I tried to wrap the process call like this:

try (Scope scope = contextSnapshot.setThreadLocals()) {
    return processor.process(entry);
}

Problems

  1. Sometimes the thread context leaks, and data is not cleared after processing.
  2. Unpredictable behavior when passing and managing the ThreadContext in distributed scenarios.

Questions

  1. How can I correctly set and clear the thread context in distributed systems like Hazelcast?
  2. Is there a better approach to propagate and manage context across JVMs?
  3. How can I avoid thread context leaks in such scenarios?

Any insights or examples would be greatly appreciated!

Upvotes: 1

Views: 51

Answers (1)

petrubear
petrubear

Reputation: 721

I'm not sure is this would be of help to you or not, but I had a similar issue I wanted to pass informaton to the logger but i have some information that goes through hazelcast and some other goes through ActiveMQ, I dont manipulate the ThreadContext myself but I rely on Slf4j MDC (https://www.slf4j.org/api/org/slf4j/MDC.html) as you can easily configure your logger to get information from it, this class has the methods getCopyOfContextMap() and setContextMap(Map<String,String> contextMap) so I just send a map as part of the message for ActiveMQ and I update it when it comes back so I have the same information on both sides on the MDC object once activemq returns.

Upvotes: 1

Related Questions