Spring Sleuth, Log4j2 different trace id in different services

Question

I'm calling from service1 to service2 with microservices configured with spring sleuth. Both microservices has same setup. I'm expecting same trace id for two services, but I'm getting different trace ids. Current output in each services.

log in service1

2019-12-07T10:51:04,219 INFO [service1,1b6f2fbf64c316fc][http-nio-8090-exec-2] c.m.p.c.MyController1: An INFO Message 1

log in service2

2019-12-07T10:51:04,976 INFO [service2,cd796e06d80b1dfd][http-nio-8091-exec-1] c.m.s.c.MyController2: An INFO Message in service 2

I'm just wandering what I'm doing wrong in following setup. kindly help.

Controller class

@RestController
@RequestMapping("/service1")
public class MyController1 {

    static final Logger LOGGER = LogManager.getLogger(MyController.class);

    @Bean
    public Sampler defaultSampler() {
        return Sampler.ALWAYS_SAMPLE;
    }

    @PostMapping("/msg")
    public String processMsg(@Valid @RequestBody final String request){

        LOGGER.info("An INFO Message 1");

        String requestURI="http://localhost:8090/service2/call1";

        HttpHeaders headers = new HttpHeaders();
        RestTemplate restTemplate = new RestTemplate();
        List<MediaType> acceptableMediaTypes = new ArrayList<MediaType>();
        acceptableMediaTypes.add(MediaType.APPLICATION_JSON);
        headers.setContentType(MediaType.APPLICATION_JSON);
        headers.setAccept(acceptableMediaTypes);
        HttpEntity<String> entity = new HttpEntity<String>(request, headers);
        ResponseEntity<String> result = restTemplate.exchange(requestURI, HttpMethod.POST, entity, String.class);
    }
}

log4j2-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Properties>
    <Property name="springAppName">${bundle:application:spring.application.name}</Property>
    <Property name="sleuthInfo">${springAppName},%X{X-B3-TraceId}</Property>
</Properties>

<Appenders>

    <Console name="Console" target="SYSTEM_OUT">
        <PatternLayout
            pattern="%style{%d{ISO8601}}{black} %highlight{%-5level }[${sleuthInfo}][%style{%t}{bright,blue}] %style{%C{1.}}{bright,red}: %msg%n%throwable"
            disableAnsi="false" />
    </Console>

    <RollingFile name="RollingFile"
        fileName="../logs/my-logger.log"
        filePattern="../logs/$${date:yyyy-MM}/my-logger-%d{-dd-MMMM-yyyy}-%i.log.gz">
        <PatternLayout>
            <pattern>%d %p %C{1.} [%t] %m%n</pattern>
        </PatternLayout>
        <Policies>
            <!-- rollover on startup, daily and when the file reaches 10 MegaBytes -->
            <OnStartupTriggeringPolicy />
            <SizeBasedTriggeringPolicy size="10 MB" />
            <TimeBasedTriggeringPolicy />
        </Policies>
    </RollingFile>
</Appenders>

<Loggers>
    <!-- LOG everything at INFO level -->
    <Root level="error">
        <AppenderRef ref="Console" />
        <AppenderRef ref="RollingFile" />
    </Root>

    <!-- LOG "com.mycompany*" at TRACE level -->
    <Logger name="com.mycompany" level="trace"></Logger>
</Loggers>

</Configuration>

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.2.1.RELEASE</version>
    <relativePath /> <!-- lookup parent from repository -->
</parent>
<groupId>com.mycompany</groupId>
<artifactId>service1</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>service1</name>
<description>service1 to communicate with service2</description>

<properties>
    <java.version>1.8</java.version>
    <spring-cloud.version>Greenwich.SR2</spring-cloud.version> <!-- Spring Cloud Sleuth -->
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
        <exclusions>
            <exclusion>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-logging</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
        <exclusions>
            <exclusion>
                <groupId>org.junit.vintage</groupId>
                <artifactId>junit-vintage-engine</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <!-- Spring Cloud Sleuth -->
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-sleuth</artifactId>
    </dependency>

    <!-- In order to use log4j2 logging library other than Logback, exclude 
        default logback from our dependencies -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
        <exclusions>
            <exclusion>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-logging</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <!-- Add log4j2 for logging -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-log4j2</artifactId>
        <version>2.2.1.RELEASE</version><!--$NO-MVN-MAN-VER$ -->
    </dependency>

    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.8.1</version>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <scope>runtime</scope>
    </dependency>

    <!-- zipkin -->
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-zipkin</artifactId>
    </dependency>

    <!-- https://mvnrepository.com/artifact/io.zipkin.brave/brave-context-log4j2 -->
    <dependency>
        <groupId>io.zipkin.brave</groupId>
        <artifactId>brave-context-log4j2</artifactId>
        <version>5.9.1</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>${spring-cloud.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
</build>

Answer 1

You're using the RestTemplate via new, you need to use it as a bean so that Sleuth can instrument it.

Answer 2

I found what I am doing wrong(I haven't use spring best practices). I am posting it and hopes someone will get benefit from it.

Wrong code:

RestTemplate restTemplate = new RestTemplate();

Correct code:

@Autowired
RestTemplate restTemplate;

@Bean
public RestTemplate getRestTemplate() {
    return new RestTemplate();
}

I initially thought I have done wrong configuration/dependencies with sleuth,log4j2. With the above fix one can use above code to tracking request in spring boot application with sleuth and log4j.