CODEWITHSUNDEEP
aws-fargatenewrelicfluent-bit
Madhukara Hiriadka
Madhukara Hiriadka

Reputation: 21

Fluentbit - New relic log integration

I am currently using fluentbit as a sidecar container to push the log into New Relic for services which is deployed as docker in ECS Fargate.

Currently log message looks as below: [37m[Info] 2022-12-9T09:08:15.346, isRequestSuccess: False, totalTime: 2603, category: <Category>, callerIpAddress: <IP address>, timeGenerated: 12/09/2022 09:08:15, region: QA, correlationId: fecdafdb-c6af-41ac-a653-ecabbc682392, method: GET, url: <Request URL>, backendResponseCode: 503, responseCode: 503, responseSize: 370, cache: none, backendTime: 1600, apiId: <API Id>, operationId: HealthCheck, productId: <Product Id>, clientProtocol: HTTP/1.1, backendProtocol: HTTP/1.1, apiRevision: 1, clientTlsVersion: 1.2, backendMethod: GET, backendUrl: <Bakend URL>, correlationId: fecdafdb-c6af-41ac-a653-ecabbc682392[0m

Its logged as a unstrcutured data and I cannot use new relic query on the specific field in the log as unstructured data.

FluentBit with following configuration:

[OUTPUT]
    name            nrlogs
    match           *
    license_key     <license-key>
    base_uri        <host>

Does anyone know how to push the logs to new relic in structured way? I have tried with few new relic parsers and that did not help me.

Any help is appreciated.

enter image description here

Upvotes: 0

Views: 486

Answers (1)

Ned L
Ned L

Reputation: 61

I'd recommend one of the following approaches:

  1. Reconfigure your service's logging framework to output the log in JSON format. New Relic can natively ingest JSON logs and all fields will be converted to attributes in New Relic, which you can use for querying/filtering/alerting.
  2. Set up a parsing rule in New Relic (using Grok expressions or plain ol' regex) to parse the logs as they're ingested into New Relic, which will result in attributes being created at ingest time. (See: https://docs.newrelic.com/docs/logs/ui-data/parsing/)
  3. Use the awesome NRQL power features aparse(..) or capture(..) to extract the relevant fields at query time. (See: https://newrelic.com/blog/how-to-relic/nrql-improvements and https://newrelic.com/blog/how-to-relic/using-regex-capture)

I hope this helps!

Upvotes: 1

Related Questions