Not able to create other fields while sending logs on new relic using newrelic_telemetry_sdk in python

Question

I am not able to send logs with other fields like " %(asctime)s-%(correlation_id)s-%(levelname)s-%(message)s- " using any handler in new relic.

I was able to see normal logs in console using newrelic_telemetry_sdk's logclient but had problem in initializing the handler. I even tried handlers such as streamHandler, SysLogHandler, FileHandler but could not send the other parameters like correlation_id and stuff with them.

Can anyone please guide as in how to and which handler to use so that I can send my asctime, correlation_id parameters along with my messages.

I am adding my some of the code below

` import logging from newrelic_telemetry_sdk import Log, LogClient from fastapi import FastAPI, Response from logging import handlers

 app = FastAPI()
 logger = logging.getLogger(__name__)
 _log_format = "%(asctime)s-%(correlation_id)s-%(levelname)s-%(message)s-%(http_method)s:%          (http_endpoint)s-%(" \

          "funcName)s-%(lineno)d]-%(user_id)s-%(client_ip)s-%(user_agent)s"

  log_client = LogClient('#Added my new relic liscence key in here')

  # handler = WSGIHandler()
  # handler = newrelic.agent.handlers.WSGIHandler()

  handler = logging.StreamHandler()
  logger.addHandler(handler)

 # handler = logging.handlers.SysLogHandler(address=('log-api.newrelic.com', 443))
 # logger.addHandler(handler)

# formatter = NewRelicContextFormatter()
# FileHandler
# fh = logging.FileHandler("/root/.newrelic/newrelic-cli.log", mode="a")
# logger.addHandler(fh)

# file = handlers.RotatingFileHandler("/root/.newrelic/newrelic-cli.log", mode="a")
# file.setFormatter(NewRelicContextFormatter())
# logger.addHandler(file)

# handler = logging.NullHandler()
# logger.addHandler(handler)`

.....The rest of my code which creates the fields mentioned in the _log_format and stores the data accordingly`

As one can see I tried many handlers for sending the _log_format fields along with the log message but could not able to see it in the logs ui of the new relic dashboard

Can anyone help me where am i going wrong!!

#Update_2:

import newrelic
import newrelic.agent
import logging

class HeaderFilter(logging.Filter):
    def __init__(self):
        super().__init__()

    def filter(self, record):
        setattr(record, "correlation_id", getattr(self, "correlation_id", None))
        setattr(record, "user_id", getattr(self, "user_id", None))
        setattr(record, "user_agent", getattr(self, "user_agent", None))

        list1 = []
        for key, value in record.__dict__.items():
            list1.append((key, value))
            newrelic.agent.add_custom_attributes(list1)
        return True
        
logger.addFilter(HeaderFilter())

#Update_3:

def global_middleware(request, call_next):

    logger.filters[0].correlation_id = request.headers.get("X-Correlation-Id", uuid.uuid4())
    logger.filters[0].user_id = request.headers.get("X-User-Id", "-")

    logger.info("START REQUEST 1234")

    return response

Answer 1

The short answer is that log formatting cannot be done with the newrelic_telemetry_sdk. It does not allow for custom formatting (as you can see here where it ignores the constructor's arguments). However, if someone REALLY wanted to create custom log formatting, they could do something like this:

import logging
import os
from newrelic_telemetry_sdk import Log, LogClient

_log_format = "%(asctime)s-%(levelname)s-%(message)s-%(funcName)s-%(lineno)d"

log_client = LogClient(os.environ["NEW_RELIC_INSERT_KEY"])
msg = "Hello World!"

record = logging.makeLogRecord({"msg": msg})
formatter = logging.Formatter(fmt=_log_format)
result = formatter.format(record)

log = Log(result)

response = log_client.send(log)
response.raise_for_status()
print("Log sent successfully!")

If, however, the objective is to pass custom attributes, this can be done with some caveats. The caveat applicable to this case is that the attributes need to exist already by the time the logger ("Log" in this case) is called.

The intent of newrelic-telemetry-sdk is to have a barebones telemetry setup that sends up telemetry to New Relic without having to use the New Relic agent. That is to say that the two are separate entities. The commands that work for the New Relic agent will not work in the telemetry SDK.

I am making certain assumptions about your setup: 1) For attributes like correlation_id, user_id, user_agent, separate logic is required to generate those (since neither the logging library or the newrelic-telemetry-sdk produce those). 2) The aforementioned logic comes before your call to log a message, so by the time "Log" is called, these attributes will exist already.

Something like this should give you the attributes along with your log message:

import os
from newrelic_telemetry_sdk import Log, LogClient

log_client = LogClient(os.environ["NEW_RELIC_INSERT_KEY"])

[Logic for correlation_id and other trace/setup variables]

# Set second argument to None to not override generated timestamp
log = Log("Hello World!", None, correlation_id=<correlation_id_var>, user_id=<user_id_var>)

response = log_client.send(log)
response.raise_for_status()
print("Log sent successfully!")