Spring boot 3 monitoring (observation, metrics)

Question

the whole observation capability around spring boot 3 is a bit of confusing. We have pure spring framework capabilities, micrometer ones and spring boot 3 on top to auto-configure stuff for us. Considering a standard @RestController application with prometheus dependency added for example, by doing absolutely nothing about observation, we already have metrics for:

• http.server.requests // incoming http requests
• http.client.requests // outgoing http requests (for RestTemplate, RestClient, etc.)
• spring.data.repository.invocations
• mongodb.driver.commands / pool
• tasks.scheduled.execution

According to docu, by overriding the corresponding default convention, we can add custom tags in runtime:

@Component
public class ServerRequestMetricsCustomizer 
    extends DefaultServerRequestObservationConvention { ... }

Though, this works only for the http.server and client requests, because they rely on a @Bean to inject their default convention, but the others (mongodb.driver and scheduled) metrics are instantiating their default convention straight away with new in MongoObservationCommandListener resp. in ScheduledMethodRunnable and for the mongodb.driver.commands metric I dont even see a default convention, which I could override to add custom tags.

Do you know the way how to add custom tags also to the other ones?

And second general question: instead of overriding these conventions, can I not inject in my production code the observation registry and somehow access the right context and add my tags? I tried it within the rest controller method, but there it does not access the http.server.requests context, rather some spring.security.http.secured.requests one

UPDATE:

found a way to configure the spring.data.repository.invocations, the mongodb.driver.commands/pool and the scheduled ones:

@Component
public class SpringDataRepositoryMetricsCustomizer
    extends DefaultRepositoryTagsProvider { ... }

resp.

@Component
public class MongoDBCommandMetricsCustomizer
    extends DefaultMongoCommandTagsProvider { ... }
@Component
public class MongoDBConnectionPoolMetricsCustomizer
    extends DefaultMongoConnectionPoolTagsProvider { ... }

resp.

@Component
public class ScheduledTaskMetricsCustomizer
    implements GlobalObservationConvention<ScheduledTaskObservationContext>
{ ... }

But now I am wondering, how can I access the data that these observations are wrapping. For example the object that is fetched by the repository methods or the one that is saved, etc. or there is no way?

Btw. the general question is still valid, cause if I could add the custom tags inside my business code, I could have an easy access to my business data and produce the custom tags without relying on access via the context objects or mongodb events

UPDATE2:

For the http.server.requests metrics I could re-use the http servlet request (which is the context carrier) attributes

For the http.client.requests metrics I could inject the http servlet request (since the context carrier is not anymore the http servlet request) and do the above, but I also could use the context to reach out to the parentObservation and read out from its map, which could be filled before hand

For the other metrics (mongodb driver and repository) I could do same as above, inject http servlet request and/or ObservationRegistry and read out from their attributes resp. map, which should be filled before hand.

The tasks scheduled metric seems to not be able to use above concept as http servlet requests is not applicable and observation registry injection here causes spring startup to report a cycle from the WebMvcObservationAutoConfiguration, so here it appears I have to rely on the ScheduledTaskObservationContext map, which should be filled before hand in the @Scheduled method by injecting there the observation registry and filling the current observation's context's map

After all, all appears working, but I am not sure how fragile this might be and whether there is no general consistent way. Maybe the consistency comes from:

• first try to use the context's map resp. context's parent observation's map (if provided), by making sure before hand it is filled
• if not, try to workaround via injecting the observation registry to reach out to the map, which again has to be pre-filled beforehand
• if not, try to workaround via injecting the http servlet request to reach out to the attributes, by making sure they were pre-filled beforehand

UPDATE3: Here is a description of how I solved it so far, hitting a limit in the micrometer framework, but still doable in an encapsulated way.

Btw. why is it so different to customize the tags for the different metrics:

• extending DefaultServerRequestObservationConvention // works like a charm
• extending DefaultClientRequestObservationConvention // works like a charm
• extending DefaultRepositoryTagsProvider // works like a charm though differs from first two ones
• extending DefaultMongoCommandTagsProvider // works like a charm though differs from first two ones
• implementing GlobalObservationConvention<ScheduledTaskObservationContext> // work only after manually adding it as part of a custom bean configuration of the ObservationRegistryCustomizer<ObservationRegistry>...observationRegistry.observationConfig().observationConvention(...)