Reputation: 93
I wrote two different script as below:
# Script 1:
current_date=$(date +"%m-%d-%Y")
logsPath="/hadoop_common/smallsite/realtime/current/spark/logs"
find $logsPath -maxdepth 1 -name "*$current_date*" -print > tmp
and
# Script 2:
current_date=$(date +"%m-%d-%Y")
logsPath="/hadoop_common/smallsite/realtime/current/spark/logs"
ls $logsPath | grep "$current_date" > tmp
sed -i "s|^|$logsPath|" tmp
First script took 24 minutes to list 2367 filePath and second script took 16 seconds to list same number of filePath.
Why this much difference? I ran them on different days, multiple times and in any order and always first script took above 20 minutes, and second script took below 20 seconds.
OS: Red Hat
UPDATE 2
find (GNU findutils) 4.4.2
I ran this script
#!/bin/bash
current_date=$(date +"%m-%d-%Y")
logsPath="/hadoop_common/smallsite/realtime/current/spark/logs"
touch resultStat
date >> resultStat
ls $logsPath | grep "$current_date" > tmp1
sed -i "s|^|$logsPath|" tmp1
date >> resultStat
find $logsPath -maxdepth 1 -type f -name "*$current_date*" -print > tmp2
date >> resultStat
strace ls "$logsPath" 2>&1 | grep -c stat >> resultStat
date >> resultStat
strace find "$logsPath" -maxdepth 1 -name "*$current_date*" -print 2>&1 | grep -c stat >> resultStat
date >> resultStat
ls -1q $logsPath | wc -l >> resultStat
date >> resultStat
files=(/hadoop_common/smallsite/realtime/current/spark/logs/*"$(date +%m-%d-%Y)"* )
date >> resultStat
resultStat content:
Thu Mar 29 19:14:28 UTC 2018
Thu Mar 29 19:14:47 UTC 2018
Thu Mar 29 19:41:26 UTC 2018
14
Thu Mar 29 19:41:44 UTC 2018
189805
Thu Mar 29 20:08:30 UTC 2018
190348
Thu Mar 29 20:08:48 UTC 2018
Thu Mar 29 20:09:06 UTC 2018
Upvotes: 2
Views: 132