Bash Script to batch-convert IP Addresses to CIDR?

Question

Ok, here's the problem.

I have a plaintext list of IP addresses that I'm blocking on my servers, growing more and more unwieldy every day (added 3000+ entries today alone).

It's already been sorted for duplicates so that's not a problem. What I'd like to do is write a script to go through it and consolidate the entries a bit better for mass blocking.

For example, take this:

2.132.35.104
2.132.79.240
2.132.99.87
2.132.236.34
2.132.245.30

And turn it into this:

2.132.0.0/16

Any suggestions on how to code that in a bash script?

UPDATE: I've worked out part-way how to do what I'm needing. Converting it to /24 is easy, as follows:

cat /usr/local/blocks/blocks.txt | while read line; do
    oc1=`echo "$line" | cut -d '.' -f 1`
    oc2=`echo "$line" | cut -d '.' -f 2`
    oc3=`echo "$line" | cut -d '.' -f 3`
    oc4=`echo "$line" | cut -d '.' -f 4`
    echo "$oc1.$oc2.$oc3.0/24" >> twentyfour.srt
done

sort -u twentyfour.srt > twentyfour.txt
rm -f twentyfour.srt
ori=`cat /usr/local/blocks/blocks.txt | wc -l`
new=`cat twentyfour.txt | wc -l`
echo "$ori"
echo "$new"

That reduced it down from 4,452 entries to 4,148 entries.

Instead of having:

109.86.9.93
109.86.26.77
109.86.55.225
109.86.70.224
109.86.87.199
109.86.89.202
109.86.95.248
109.86.100.19
109.86.110.43
109.86.145.216
109.86.152.86
109.86.155.238
109.86.156.54
109.86.187.91
109.86.228.86
109.86.234.51
109.86.239.61

I now have:

109.86.100.0/24
109.86.110.0/24
109.86.145.0/24
109.86.152.0/24
109.86.155.0/24
109.86.156.0/24
109.86.187.0/24
109.86.228.0/24
109.86.234.0/24
109.86.239.0/24
109.86.26.0/24
109.86.55.0/24
109.86.70.0/24
109.86.87.0/24
109.86.89.0/24
109.86.9.0/24
109.86.95.0/24

All well and good. BUT, there's 17 entries from the 109.86.. area. In a case where the first 2 octets match more than say 5 entries on /24, I'd like to reduce that to /16.

That's where I'm stuck.

UPDATE 2:

For Steve: Here's the block list for today. And here's the result so far. Apparently it's not removing the near-duplicate entries from twentyfour that are in sixteen.

Answer 1

I wish I could tell you this is a simple filter. However, all of the 2.0.0.0/8 network is registered to RIPE NCC. There's just way too many different ranges of blocked IP addresses, its easier to just narrow down the scope of visitors you do want versus what you don't want.

You could also use various tools you can use to block attacks automatically.

Map to identify which is which. https://www.iana.org/numbers Here's a script I just made for you. Then you can create the major block lists for each of the primary registries. Afrinic, Lacnic, Apnic, Ripe, and Arin. create_tables_by_registry.sh

Just run this script... Then run the following registry.sh files. (E.g; ripe.sh)

#!/bin/bash
# Author: Steve Kline
# Date: 03-04-2014
# Designed and tested to run on properly on CentOS 6.5
#Grab Updated IANA Address Space Assignments only if Newer Version
    wget -N https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.txt
assigned=ipv4-address-space.txt
arrayregistry=( afrinic apnic arin lacnic ripe )
for registry in "${arrayregistry[@]}"
do
#Clean up the ipv4-address-space.txt file and keep useable IPs
grep "$registry" $assigned | sed 's/\/8/\.0\.0\.0\/8/g'| colrm 15 > $registry-tmp1.txt
ip=($(cat $registry-tmp1.txt))
echo "#!/bin/bash" > $registry.sh
for ip in "${ip[@]}"
    do
    echo $ip | sed -e 's/"   "//g'  > $registry-tmp2.txt
    #INSERT OR MODIFY YOUR COMPATIBLE FIREWALL RULES HERE
    #This section creates the country to block.
    echo "iptables -A INPUT -s $ip -j DROP" >> $registry.sh
    chmod +x $registry.sh
done
    rm $registry-tmp1.txt -f
    rm $registry-tmp2.txt -f
done

Ok! Well I'm back, a little insane here and a little nutty there... I think I helped figure this out for you. I'm sure you can piece together a modification to better fit your needs.

#MODIFY FOR YOUR LIST OF IP ADDRESSES
BADIPS=block.ip
twentyfour=./twentyfour.ips #temp file for all IPs converted to twentyfour net ids
sixteen=./sixteen.ips   #temp file for sixteen bit
twentyfourlst1=./twentyfour1.txt    #temp file for 24 bit IDs
twentyfourlst2=./twentyfour2.txt    #temp file for 24 bit IDs filtered by 16 bit IDs that match
sixteenlst=./sixteen.txt    #temp file for parsed sixteenbit
#MODIFY FOR YOUR OUTPUT OF CIDR ADDRESSES
finalfile=./blockips.list   #Final file post-merge

cat $BADIPS | while read line; do
oc1=`echo "$line" | cut -d '.' -f 1`
oc2=`echo "$line" | cut -d '.' -f 2`
oc3=`echo "$line" | cut -d '.' -f 3`
oc4=`echo "$line" | cut -d '.' -f 4`
echo "$oc1.$oc2.$oc3.0/24" >> $twentyfour
echo "$oc1.$oc2.0.0/16" >> $sixteen
done
awk '{i=1;while(i <= NF){a[$(i++)]++}}END{for(i in a){if(a[i]>4){print i,a[i]}}}' $sixteen | sed 's/ [0-9]\| [0-9][0-9]\| [0-9][0-9][0-9]//g' > $sixteenlst
sort -u $twentyfour > twentyfour.txt
# THIS FINDS NEAR DUPLICATES MATCHING FIRST TWO OCTETS
cat $sixteenlst | while read line; do
   oc1=`echo "$line" | cut -d '.' -f 1`
   oc2=`echo "$line" | cut -d '.' -f 2`
   oc3=`echo "$line" | cut -d '.' -f 3`
   oc4=`echo "$line" | cut -d '.' -f 4`
   grep "\b$oc1.$oc2\b" twentyfour.txt >> duplicates.txt    
done
#THIS REMOVES THE NEAR DUPLICATES FROM THE TWENTYFOUR FILE
fgrep -vw -f duplicates.txt twentyfour.txt > twentyfourfinal.txt
#THIS MERGES BOTH RESULTS
cat twentyfourfinal.txt $sixteenlst > $finalfile
sort -u $finalfile
ori=`cat $BADIPS | wc -l`
new=`cat $finalfile | wc -l`
echo "$ori"
echo "$new"
#LAST MIN CLEANUP
rm -f $twentyfour $twentyfourlst $sixteen $sixteenlst duplicates.txt twentyfourfinal.txt

Going Back to fix: I noted a problem... Originally unsuccessful. `grep "$oc1.$oc1" twentyfour.txt > duplicates.txt

For Example: The old script had bad results with this test IP range... the updated version now above... Does exactly as its intended. match the octet exactly.. and not a similar.

192.168.1.1
192.168.2.50
192.168.5.23
192.168.14.10
192.168.10.5
192.168.24.25
192.165.20.10
10.192.168.30
5.76.10.20
5.76.20.30
5.76.250.10
5.76.34.10
5.76.50.30
95.76.30.1    - Old script matched this to 5.76
20.20.5.5
20.20.10.10
20.20.16.50
20.20.205.20
20.20.60.20
205.20.16.20 - not a  problem
20.205.150.150 - Old script matched this to 20.20
220.20.16.0 - Also failed without adding -w parameter to the last grep to only match exact strings.