CODEWITHSUNDEEP
open-telemetryopen-telemetry-collector
zamentali
zamentali

Reputation: 330

OpenTelemetry collector - redact log content

I'm trying to redact certain sensitive information bits that are in the log files before exporting using the Open Telemetry collector. Here's my json log (I'm using Monologue and have applied json formatting to make it easier to configure json_parser on the collector config).

Problem - I can't seem to figure out how to redact the value "XnHyH4QALxXnQwmvw7XtB2brs63K4pby" from the log below.

See my yaml config below.

processors:
  attributes:
    actions:
      - key: body
        pattern: 'XnHyH4QALxXnQwmvw7XtB2brs63K4pby'
        action: hash

  redaction:
    blocked_values:
      - "XnHyH4QALxXnQwmvw7XtB2brs63K4pby"

both the hash attribute processor and redaction processor don't work. See log below.

How can I achieve the redaction?

{
   "body":"Request full data: {\n    \"username\": \"user\",\n    \"key\": \"XnHyH4QALxXnQwmvw7XtB2brs63K4pby\",\n    \"operator\": \"123\",\n    \"no\": \"123\",\n    \"units\": 20\n}",
   "attributes":{
      "host_and_client":{
         
      },
      "channel":"production",
      "context":{
         
      },
      "datetime":"2023-06-28T12:51:56.962453+03:00",
      "extra":{
         
      },
      "level":200,
      "level_name":"INFO"
   }
}

Upvotes: 2

Views: 1541

Answers (3)

Allenile
Allenile

Reputation: 462

I successfully resolved a similar issue where I needed to redact all the tokens in my logs. Thanks to @zamentali's answer, I used a regex, and it worked perfectly:

processors:
  transform/redact_sensitive_info:
    error_mode: ignore
    log_statements:
    - context: log
      statements:
      - replace_pattern(body, "token\\=[^\\s]*(\\s?)", "token=REDACTED")
      - replace_pattern(attributes["message"], "token\\=[^\\s]*(\\s?)", "token=REDACTED")

service:
  pipelines:
    logs:
      receivers: [xyz]
      processors: [transform/redact_sensitive_info, xyz]
      exporters: [xyz]

Upvotes: 0

ppal
ppal

Reputation: 31

I have been struggling in log transformation. The log looks like this.

2024-03-11T21:04:41.411025006Z stdout F {"time": "2024-03-11T21:04:41+00:00", "upstream_namespace":"system-monitoring", "remote_user": "sample-user"}

So far I have tried the following but none seems to work, any suggestions?

processors:
  attributes/upsert:
   actions:
   - key: upstream_namespace
     action: upsert
     value: "REDACTED_NS"
  transform:  
   log_statements:
   - context: log
     statements:
      - replace_all_patterns(attributes,"value","upstream_namespace", "REDACTED_NS")
      - replace_all_patterns(attributes,"key","upstream_namespace", "REDACTED_NS")
      - replace_match(attributes["upstream_namespace"], "*" , "REDACTED_NS")
      - replace_match(attributes["upstream_namespace"], "system-monitoring" , "REDACTED_NS")
      - delete_key(attributes,"upstream_namespace")
      - delete_key(resource.attributes,"upstream_namespace")
      - replace_all_patterns(attributes["upstream_namespace"],"value","upstream_namespace", "REDACTED_NS")
      - replace_all_patterns(attributes["upstream_namespace"],"value","system-monitoring", "REDACTED_NS")

The attribute/upsert however adds REDACTED_NS value along with the original.

upstream_namespace: REDACTED_NS
                    system-monitoring

Upvotes: 0

zamentali
zamentali

Reputation: 330

I managed to solve this using transform processor. Incase someone is wondering how to do it in future, here's how!

processors:
  transform/redact_sensitive_info:
    error_mode: ignore
    log_statements:
    - context: log
      statements:
      - replace_pattern(attributes["message"], "XnHyH4QALxXnQwmvw7XtB2brs63K4pby", "********")
      - replace_pattern(body, "XnHyH4QALxXnQwmvw7XtB2brs63K4pby", "********")


service:
  pipelines:
    logs:
      receivers: [xyz]
      processors: [transform/redact_sensitive_info, xyz]
      exporters: [xyz]

PS - In my case, I have the log in the body section as well as in the attributes.message that's why there are two replace pattern statements. Adjust to suite your needs.

Transform processor documentation is here -

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md

Upvotes: 2

Related Questions