CODEWITHSUNDEEP
multithreadingbashtcpdump
dobbs
dobbs

Reputation: 1043

speed up bash script with multithreading?

I've got a bash script that i put together to merge multiple packet captures based on a common filter. I'm running daemonlogger on the back end and it rolls pcap files based on size so its difficult to get the whole picture sometimes as the data i'm looking for may be in one pcap file and the rest in another.. The biggest gripe i have is the inability to speedup this process. It can only process one pcap at a time. Does anyone have any recommendations on how to speed this up with multiple subprocesses or multiple threads? 

#!/bin/bash
echo '[+] example tcp dump filters:'
echo '[+] host 1.1.1.1'
echo '[+] host 1.1.1.1 dst port 80'
echo '[+] host 1.1.1.1 and host 2.2.2.2 and dst port 80'
echo 'tcpdump filter:'
read FILTER
cd /var/mycaps/
DATESTAMP=$(date +"%m-%d-%Y-%H:%M")
# make a specific folder to drop the filtered pcaps in
mkdir /var/mycaps/temp/$DATESTAMP
# iterate over all pcaps and check for an instance of your filter
for file in $(ls *.pcap); do
        tcpdump -nn -A -w temp/$DATESTAMP/$file -r $file $FILTER
        # remove empty pcaps that dont match
        if [ "`ls -l temp/$DATESTAMP/$file | awk '{print $5}'`" = "24" ]; then
                rm -f "temp/$DATESTAMP/$file"
        fi
done
echo '[+] Merging pcaps'
# cd to your pcap directory 
cd /var/mycaps/temp/${DATESTAMP}
# merge all of the pcaps into one file and remove the seperated files
mergecap *.pcap -w merged.pcap
rm -f original.*
echo "[+] Done. your files are in $(pwd)"

Upvotes: 1

Views: 257

Answers (1)

chepner
chepner

Reputation: 531868

Run the body of the loop in the background, then wait for all the background jobs to complete before continuing.

max_jobs=10   # For example
job_count=0
for file in *.pcap; do   # Don't iterate over the output of ls
    (tcpdump -nn -A -w temp/"$DATESTAMP"/"$file" -r "$file" $FILTER
    # remove empty pcaps that don't match. Use stat to get the file size
    if [ "$(stat -c "%s")" = 24 ]; then
            rm -f "temp/$DATESTAMP/$file"
    fi
    ) &
    job_count=$((job_count+1))
    if [ "$job_count" -gt "$max_jobs" ]; then
        wait
        job_count=0
    fi
done
wait

Upvotes: 2

Related Questions