CODEWITHSUNDEEP
performancebashcurlperformance-testing
BaltoStar
BaltoStar

Reputation: 8977

bash loop with curl evidencing non-linear scaling of response times

I wrote this simple Bash script to detect incidence of error-pages: 

date; 
iterations=10000; 
count_error=0; 
count_expected=0; 
for ((counter = 0; counter < iterations; ++counter)); do 
  if curl -s http://www.example.com/example/path | grep -iq error; 
  then 
    ((count_error++)); 
  else 
    ((count_expected++)); 
  fi; 
sleep 0.1; 
done; 
date;
echo count_error=$count_error count_expected=$count_expected

I'm finding total execution-time does not scale linearly with iteration count. 10 iterations 00:00:12, 100 in 00:01:46, 1000 in 00:17:24, 10000 in ~50 mins, 100000 in ~10 hrs

Can anyone provide insights into the non-linearity and/or improvements to the script? Is curl unable to fire requests at rate of 10/sec? Is GC having to periodically clear internal buffers filling up with response text ?

Upvotes: 0

Views: 440

Answers (1)

Michael Jaros
Michael Jaros

Reputation: 4681

Here are a few thoughts:

  • You are not creating 10 requests per second here (as you stated in the question), instead you are running the requests sequentially, i.e. as many per second as possible.
  • The ; at the end of each line is not required in Bash.
  • When testing your script from my machine against a different URL, 10 iterations take 3 seconds, 100 take 31 seconds, and 1000 take 323 seconds, so the execution time scales linearly in this range.
  • You could try using htop or top to identify performance bottlenecks on your client or server.
  • The apache benchmark tool ab is a standard tool to benchmark web servers and available on most distributions. See manpage ab(1) for more information.

Upvotes: 1

Related Questions