CODEWITHSUNDEEP
bashfor-looptcpdump
dobbs
dobbs

Reputation: 1043

parallel execution bash for loop

i've put together a simple little bash script that iterates through 100-200+ 2GB packet captures (from daemonlogger), which prompts the user for the filter to match on in tcpdump... and then compiles all of the packets from each individual capture into 1 merged cap. One thing i've run into is i'd like for the execution to happen faster by searching multiple packet captures at once... but without just simply backgrounding with &. (tried that and it basically brought the system to its knees trying to load tons of 2GB pcaps into memory. lol oops.) Can anyone show me how within a for-loop you can say "i want to run two or three iterations at a time for each pass." etc?

#!/bin/bash
echo '[+] example tcp dump filters:'
echo '[+] host 1.1.1.1'
echo '[+] host 1.1.1.1 dst port 80'
echo '[+] host 1.1.1.1 and host 2.2.2.2 and dst port 80'
echo 'tcpdump filter:'
read FILTER
cd /var/packet_recorder/
DATESTAMP=$(date +"%m-%d-%Y-%H:%M")
# make a specific folder to drop the filtered pcaps in
mkdir /var/packet_recorder/temp/$DATESTAMP
# iterate over all pcaps and check for an instance of your filter
for file in $(ls *.pcap); do
        tcpdump -nn -A -w temp/$DATESTAMP/$file -r $file $FILTER
        # remove empty pcaps that dont match
        if [ "`ls -l temp/$DATESTAMP/$file | awk '{print $5}'`" = "24" ]; then
                rm -f "temp/$DATESTAMP/$file"
        fi
done
echo '[+] Merging pcaps'
# cd to your pcap directory 
cd /var/packet_recorder/temp/${DATESTAMP}
# merge all of the pcaps into one file and remove the seperated files
mergecap *.pcap -w merged.pcap
rm -f InternetBorder.*
echo "\[\+\] Done. your files are in $(pwd)"

Upvotes: 0

Views: 214

Answers (1)

yaegashi
yaegashi

Reputation: 1520

Recently I've learned from this question about using GNU Parallel or xargs -P in GNU Findutils for such a problem.

Using xargs -P (assuming there's no space in paths)

# iterate over all pcaps and check for an instance of your filter
# process up to 3 files at a time
ls *.pcap | xargs -n1 -P3 -I{} tcpdump -nn -A -w temp/$DATESTAMP/{} -r {} $FILTER

# remove empty pcaps that dont match (remove files whose size is 24)
wc -c temp/$DATESTAMP/*.pcap | head -n -1 |
while read size path; do
    if [[ "$size" = 24 ]]; then
        rm -f "$path"
    fi
done

Upvotes: 1

Related Questions