text processing for IPv4 dotted decimal notation conversion to /8 or /16 format

Question

I have an input file that contains a list of ip addresses and the ip_counts(some parameter that I use internally.)The file looks somewhat like this.

202.124.127.26  2135869
202.124.127.25  2111217
202.124.127.17  2058082
202.124.127.16  2014958
202.124.127.20  1949323
202.124.127.24  1933773
202.124.127.27  1932076
202.124.127.22  1886466
202.124.127.18  1882955
202.124.127.21  1803528
202.124.127.23  1786348
119.224.129.200  1776592
119.224.129.211  1639325
202.124.127.19  1479198
119.224.129.201  1145426
202.49.175.110  1133354
119.224.129.210  1119525
68.232.45.132  1085491
119.224.129.209  1015078
131.203.3.8   857951
202.162.73.4   817197
207.123.58.125   785326
202.7.6.18   762603
117.121.253.254   718022
74.125.237.120   710448
68.232.44.219   693002
202.162.73.2   671559
205.128.75.126   611301
119.161.91.17   604393
119.224.129.202   559930
8.27.241.126   528862
74.125.237.152   517516
8.254.9.254   514341

As you can see the ip addresses themselves are unsorted.So I use the sort command on the file to sort the ip addresses as below

cat address_count.txt | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > sorted_address.txt

Which gives me an output with ip addresses in the sorted order.The partial output of that file is shown below.

4.23.63.126    15731
4.26.254.254   320705
4.27.8.254    25174
8.12.129.50   176141
8.12.223.125    11800
8.19.32.65    15854
8.19.240.53    11013
8.19.240.70    11915
8.19.240.72    31541
8.19.240.73    23304
8.20.213.28    96434
8.20.213.32   108191
8.20.213.34   170058
8.20.213.39    23512
8.20.213.41    10420
8.20.213.61    24809
8.26.195.253    28568
8.27.152.253   104446
8.27.233.125   115856
8.27.235.126    16102
8.27.235.254    25628
8.27.238.254   108485
8.27.240.125   169262
8.27.241.126   528862
8.27.241.252   197302
8.27.248.125    14926
8.254.9.254   514341
12.129.210.71    89663
15.192.45.21    20139
15.192.45.26    35265
15.193.0.148    10313
15.193.113.29    40318
15.201.49.136    14243
15.240.238.52    57163
17.250.248.95    28166
23.33.125.13    19179
23.33.125.37    17953
31.151.163.60    72709
38.99.42.37   192356
38.99.68.180    41251
38.99.68.181    10272
38.104.237.74    74012
38.108.112.103    37034
38.108.112.115    69698
38.108.112.121    92173
38.108.112.122    99230
38.112.63.238    39958
38.119.130.62    42159
46.4.28.22       19769

Now I want to parse the file given above and convert it to aaa.bbb.ccc.0/8 format and aaa.bbb.0.0/16 format and I also want to count the number of occurences of the ip's in each subnet.I want to do this using bash.I am open to using sed or awk.How do I achieve this.

For example

8.19.240.53    11013
8.19.240.70    11915
8.19.240.72    31541
8.19.240.73    23304
8.20.213.28    96434
8.20.213.32   108191
8.20.213.34   170058
8.20.213.39    23512
8.20.213.41    10420
8.20.213.61    24809

The about input portion should produce 8.19.240.0/8 and 8.20.213.0/8 and similarly for /16 domains.I also want to count the occurences of machines in the subnet. For example In the above output this subnet should have the count 4 in the next column beside it.It should also add the already displayed count.i.e (11013 + 11915 + 31541 + 23304) in another column.

8.19.240.0/8 4 (11013 + 11915 + 31541 + 23304) 8.20.213.0/8 6 (96434 + 108191 + 170058 + 23512 + 10420 + 24809)

It would be great if someone could suggest some way to achieve this.

Answer 1

This might work for you:

awk '{a=$1;sub(/\.[^.]*$/,"",a);ac[a]++;at[a]+=$2};END{for(x in ac)print x".0/8",ac[x],at[x]}' file

This prints the '0/8 addresses to get the 0/16 duplicate the code i.e. b=a;sub(/\.[^.]*$/,"",b);ba[b]++ etc, etc.

Answer 2

The main problem here is that without having the routing table from the individual moments the packets arrived, you have no idea what netblock they were originally in. Sure, you can put them in the class-full blocks they would be in, in a class-full routing situation, but all that will give you is a nice presentation (and, admittedly, a shorter file).

Furthermore, your example looks a bit broken. You have a bunch of IP addresses in 8.0.0.0/8 and you are aggregating them into what looks like /24 routes and presenting them with a /8 at the end.

Nonetheless, in awk you can use sub() to do text replacement (or you can use index to find occurrences of ., or you can use split to split at dots). It should be relatively easy to go from that to "drop last digit, add the string "0/24" and use that as a key to update an IP-count and a hit-count dictionary, then drop the last two octets and the slash, replace with "0.0/16" and do the same" (all arrays in awk are associative arrays, so essentially dicts). No need to sort in advance, when you loop through the result, you'll get the keys in a random order, but on average there will be fewer of them, so sorting afterwards will be cheaper.

I seem to not have an awk at hand, so I cannot give you a code example.