Debugging triggered modsecurity rule hits

Question

TL;DR:

How do I debug a hit on the mod-security rules to identify the part of the request triggering the rule?.

The situation

I have a site runing on an environment with modsecurity and Free OWASP ModSecurity Core Rule Set (CRS) which I actually like the idea of. After spending a lot of time getting the same plugins and configuration running on my dev machine, I have an issue debugging.

I have some editors (ckeditor 4) to allow admins to edit welcome pages, email templates, etc. which produces html that is either showed in the browser, or sent in an email.

The issue is that I keep having data that is triggering different OWASP rules, resulting in a 403, which is annoying at best and not very user friendly.

Attempting to debug

For instance, I get a hit like this in the log:

[Sat Jul 11 20:12:47.827894 2020] [:error] [pid 9504:tid 2352] [client 127.0.0.1:53355] [client 127.0.0.1] ModSecurity: Warning. Pattern match "(?i)(?:\\\\W|^)(?:javascript:(?:[\\\\s\\\\S]+[=\\\\\\\\(\\\\[\\\\.<]|[\\\\s\\\\S]*?(?:\\\\bname\\\\b|\\\\[ux]\\\\d))|data:(?:(?:[a-z]\\\\w+\\\\/\\\\w[\\\\w+-]+\\\\w)?[;,]|[\\\\s\\\\S]*?;[\\\\s\\\\S]*?\\\\b(?:base64|charset=)|[\\\\s\\\\S]*?,[\\\\s\\\\S]*?<[\\\\s\\\\S]*?\\\\w[\\\\s\\\\S]*?>))|@\\\\W*?i\\\\W*?m\\\\W*?p\\\\W*? ..." at ARGS:sluttekst. [file "C:/udvikling/xampp/apache/conf/extra/owasp-modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "259"] [id "941170"] [msg "NoScript XSS InjectionChecker: Attribute Injection"] [data "Matched Data: \\x22data:image/png; found within ARGS:tekst: <p><span style=\\x22font-size: 14px;\\x22><span style=\\x22font-family: verdana,geneva,sans-serif;\\x22>her er der noget tekst som er fremtr&aelig;dende. Omr&aring;der obliver v&aring;de.Privat. Kvalitet b&oslash;r v&aelig;rds&aelig;ttes, hvis du vil.<br />\\x0d\\x0a<br />\\..."]  [hostname "devpc.local"] [uri "/z.php"] [unique_id "XwoBH4TwpS3bcEzonWuZAwAAji0"], referer: https://depc.local/z.php

Since the pattern is truncated, I looked up the config file in the local apache config (REQUEST-941-APPLICATION-ATTACK-XSS.conf) and copied the string that was matched

<p><span style=\\x22font-size: 14px;\\x22><span style=\\x22font-family: verdana,geneva,sans-serif;\\x22>her er der noget tekst som er fremtr&aelig;dende. Omr&aring;der obliver v&aring;de.Privat. Kvalitet b&oslash;r v&aelig;rds&aelig;ttes, hvis du vil.<br />\\x0d\\x0a<br />\\...

... along with what I assume is a regex pattern ...

(?i)(?:\W|^)(?:javascript:(?:[\s\S]+[=\\\(\[\.<]|[\s\S]*?(?:\bname\b|\\[ux]\d))|data:(?:(?:[a-z]\w+\/\w[\w+-]+\w)?[;,]|[\s\S]*?;[\s\S]*?\b(?:base64|charset=)|[\s\S]*?,[\s\S]*?<[\s\S]*?\w[\s\S]*?>))|@\W*?i\W*?m\W*?p\W*?o\W*?r\W*?t\W*?(?:\/\*[\s\S]*?)?(?:[\"']|\W*?u\W*?r\W*?l[\s\S]*?\()|\W*?-\W*?m\W*?o\W*?z\W*?-\W*?b\W*?i\W*?n\W*?d\W*?i\W*?n\W*?g[\s\S]*?:[\s\S]*?\W*?u\W*?r\W*?l[\s\S]*?\("

... into a tool like regexr.com to see the actual match. But no such luck.

Non-optimal workaround

I have already taken one such editor and converted the sender to base64 encode the tekst before sending it to the server, and then base64decoding it again... I don't want to do that for all the editors I have...

Where to from here?

I'd prefer to filter out all the invalid code (I already found out that the scandinavian character å htmlencoded to å is caught by the REQUEST-932-APPLICATION-ATTACK-RCE plugin, so I will have to make sure they are sent as UTF-8 characters instead)

Answer 1

CRS dev-on-duty here. First of all, this was (it's an old question, but I still would like to help) your request pattern that matched the rule 941170, the rule which you already found out: https://regex101.com/r/lq7rWf/1. You find this match behind Matched data: in your posted log line.

I can not help you tweak your editors to not send such html requests, but I can help you tune your CRS so that it does not match legitimate requests. For example to resolve the false positive you mentioned, you could tune the CRS like this: You could remove the argument sluttekst from this rule 941170 as it obviously contains legitimate traffic that triggers this CRS rule. You have to put this after your CRS include:

SecRuleUpdateTargetById 941170 "!ARGS:sluttekst"

Next, I think it would be helpful for your users when you begin running the CRS with a higher threshold so that not every matched rule blocks the user with a 403. You can raise the inbound_anomaly_score_threshold in the crs-setup.conf by uncommenting the rule and raising this threshold. Then you tune your CRS installation and after tuning it you can lower the threshold again, so that it blocks.

If you would like to learn more about how to roll out the CRS and how to tune and how to go from a higher threshold to a blocking threshold, I highly recommend this tutorial from our CRS co-lead Christian: https://www.netnea.com/cms/apache-tutorial-8_handling-false-positives-modsecurity-core-rule-set/ or the CRS documentation in general: https://coreruleset.org/docs/

Answer 2

The log entry indicates: Matched Data: \\x22data:image/png

But there is no image/png in the matched string.

Perhaps the string itself was truncated in the error log.

You could test with small pieces, starting with just <p>, to find what is blocked.

In my experience, rule 941 blocks ALL valid HTML in form fields, not just malicious code. Its 403 Forbidden response is very unfriendly to users as well. So I disabled that rule and I'm sanitizing html within the app instead of via ModSecurity.

Answer 3

an idea to work around here is:

• set CKeditor to work with / accept limited set of HTML tags using built-in ACF,
• whitelist all html tags defined in CKeditor's ACF in CRS once again,
• assure the webserver is using UTF-8,
• let modsecurity block anything else.

That's theory, implementation follows some day ... I dislike, however, an idea of outsourcing/repeat modsecurity logic to ACF, yet another layer in a stack.