CODEWITHSUNDEEP
bashlogginggrep
firepro20
firepro20

Reputation: 371

Get unique entries in a log file using grep

I have the following logfile

2016-05-02 11:37:32,211 (glastopf.glastopf) 190.244.96.69 requested GET /phpMyAdmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:33,065 (glastopf.glastopf) 190.244.96.69 requested GET /pma/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:33,990 (glastopf.glastopf) 190.244.96.69 requested GET /myadmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:55:07,672 (glastopf.glastopf) 141.8.83.213 requested GET / on 04680bb24791:80
2016-05-02 11:55:07,719 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:07,843 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:09,195 (glastopf.glastopf) 141.8.83.213 requested POST /index on 04680bb24791:80
2016-05-02 11:55:09,233 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:09,269 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:10,899 (glastopf.glastopf) 141.8.83.213 requested POST /index on 04680bb24791:80
2016-05-02 11:55:10,935 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:10,970 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:13,307 (glastopf.glastopf) 141.8.83.213 requested POST /comments on 04680bb24791:80
2016-05-02 11:55:13,359 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:13,378 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:28,106 (glastopf.glastopf) 141.8.27.157 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,193 (glastopf.glastopf) 31.13.102.123 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,205 (glastopf.glastopf) 141.8.27.157 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:28,381 (glastopf.glastopf) 173.252.120.102 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,512 (glastopf.glastopf) 141.8.27.157 requested GET /favicon.ico on 04680bb24791:80

I want to get unique entries, discarding the first three attributes i.e date time and glastopf glastopf. I want to get the unique entrie from the IP onwards. I have the following grep command:

sudo grep 'GET\|POST' /home/tsec/prototype/logs/glastopf.log | \
     sort -k4,4 | tac | sort -k4,4 | sort -k1,2 | \
     tail -n 20 > /home/tsec/prototype/logs/extractedlogs/glastopfresult.log

There must be something wrong in the sort

Upvotes: 1

Views: 104

Answers (1)

heemayl
heemayl

Reputation: 42047

This is a task for sort:

sort -uk4 file.log

This will get the first line when multiple lines are identical starting from 4th field, if you want the last entry use tac to reverse the lines and do the operation:

tac file.log | sort -uk4

Example:

$ cat file.txt
2016-05-02 11:37:32,211 (glastopf.glastopf) 190.244.96.69 requested GET /phpMyAdmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:33,065 (glastopf.glastopf) 190.244.96.69 requested GET /pma/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:33,990 (glastopf.glastopf) 190.244.96.69 requested GET /myadmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:55:07,672 (glastopf.glastopf) 141.8.83.213 requested GET / on 04680bb24791:80
2016-05-02 11:55:07,719 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:07,843 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:09,195 (glastopf.glastopf) 141.8.83.213 requested POST /index on 04680bb24791:80
2016-05-02 11:55:09,233 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:09,269 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:10,899 (glastopf.glastopf) 141.8.83.213 requested POST /index on 04680bb24791:80
2016-05-02 11:55:10,935 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:10,970 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:13,307 (glastopf.glastopf) 141.8.83.213 requested POST /comments on 04680bb24791:80
2016-05-02 11:55:13,359 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:13,378 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:28,106 (glastopf.glastopf) 141.8.27.157 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,193 (glastopf.glastopf) 31.13.102.123 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,205 (glastopf.glastopf) 141.8.27.157 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:28,381 (glastopf.glastopf) 173.252.120.102 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,512 (glastopf.glastopf) 141.8.27.157 requested GET /favicon.ico on 04680bb24791:80

$ sort -uk4 file.txt
2016-05-02 11:55:28,512 (glastopf.glastopf) 141.8.27.157 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:28,106 (glastopf.glastopf) 141.8.27.157 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,205 (glastopf.glastopf) 141.8.27.157 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:07,843 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:07,672 (glastopf.glastopf) 141.8.83.213 requested GET / on 04680bb24791:80
2016-05-02 11:55:07,719 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:13,307 (glastopf.glastopf) 141.8.83.213 requested POST /comments on 04680bb24791:80
2016-05-02 11:55:09,195 (glastopf.glastopf) 141.8.83.213 requested POST /index on 04680bb24791:80
2016-05-02 11:55:28,381 (glastopf.glastopf) 173.252.120.102 requested GET / on 04680bb24791:80
2016-05-02 11:37:33,990 (glastopf.glastopf) 190.244.96.69 requested GET /myadmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:32,211 (glastopf.glastopf) 190.244.96.69 requested GET /phpMyAdmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:33,065 (glastopf.glastopf) 190.244.96.69 requested GET /pma/scripts/setup.php on 04680bb24791:80
2016-05-02 11:55:28,193 (glastopf.glastopf) 31.13.102.123 requested GET / on 04680bb24791:80

Upvotes: 4

Related Questions