CODEWITHSUNDEEP
regexreplacesublimetext
Typewar
Typewar

Reputation: 983

How can I remove lines that do not contain a certain string?

I got a brute-force attack on my website. The log files are 1.4 GB large (4338.995 lines). How can I remove lines that do not contain a certain string in Sublime?

It contains both normal users and brute-force attacks from two different IP addresses (personal info and IP addresses have been changed):

163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /box1_rhs/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /isaac_working/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
66.29.166.6 - - [28/Apr/2017:13:00:06 +0200] "GET /index.php HTTP/1.1" 200 2898 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /title_bykergrove_red/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /games_title/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
66.29.166.6 - - [28/Apr/2017:13:00:06 +0200] "GET /info.php HTTP/1.1" 200 565 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /box1_btm/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /coast_gal_bamburgh-thumb/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /games_pic2/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /coast_gal_tentsmuir-thumb/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /pannel_bot/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /but_go_red/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /coast_gal_badbea-thumb/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /top_girl/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
61.68.207.144 - - [28/Apr/2017:13:00:06 +0200] "GET /s/ HTTP/1.1" 200 9707 "http://google.com/search?q=s06e13" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /pannel_poles_bottom/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /box2_rhs/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /watch_animals/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /pets_pic4/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /boy/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /box2_top/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /pets_title/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /coast_gal_whitby-thumb/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:06 +0200] "HEAD /box2_schoolsout_paramedic/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:07 +0200] "HEAD /rws_sign/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
12.180.245.229 - - [28/Apr/2017:13:00:07 +0200] "GET /browse.php HTTP/1.1" 200 3819 "https://www.google.com.au/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36"
163.33.74.115 - - [28/Apr/2017:13:00:07 +0200] "HEAD /box2_btm/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:07 +0200] "HEAD /serious_amazon/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:07 +0200] "HEAD /box3_noproblem_textbullying/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:07 +0200] "HEAD /ramblings12_home/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"
163.33.74.115 - - [28/Apr/2017:13:00:07 +0200] "HEAD /chain_cat/ HTTP/1.1" 404 157 "-" "DirBuster-0.12 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)"

I want to remove lines not containing "163.33.74.115". With search & replace I used:

^((?!163\.33\.74\.115).)*$

But the program didn't do anything even though everything else than the IP was highlighted. How can I do this?

Upvotes: 31

Views: 32755

Answers (2)

Pedro Lobito
Pedro Lobito

Reputation: 99001

For Sublime use:

1 - CTRL+H
2 - Click Regular Expressions (check ps below)
3 - Find What: ^163.33.74.115.*\n or ^(?!163.33.74.115).*\n for inverted matches
4 - Replace With: blank
5 - Click Replace All

enter image description here

GREP ANSWER:

The answer above should work fine, but I'd rather use grep, which is bundled with linux and mac, for windows get it here, i.e.:

1 - All lines except the ones containing 163.33.74.115:

grep -v 163.33.74.115 original.log > attack.log

2 - All lines containing 163.33.74.115:

grep 163.33.74.115 original.log > attack.log

Options:

-v, --invert-match        select non-matching lines

Upvotes: 42

Cristian Diaconescu
Cristian Diaconescu

Reputation: 35701

A quicker option is to just use the 'Find All' option: It selects all the matches for you, so you can copy them.

  1. Ctrl+F
  2. Match the lines you want using simple, positive-logic regex: .*163.33.74.115.*
  3. Click "Find All"
  4. Ctrl+C > Open New document > Ctrl+V

The advantage here is that you don't have to remember the regex syntax for negative lookahead -- which is even trickier if you're trying to match on something not at the beginning of the line.

Upvotes: 32

Related Questions