CODEWITHSUNDEEP
bashcurl
Kariem
Kariem

Reputation: 4873

Running curl in a loop results in unexpected output

In curl, I do something like this to check if a URL is available:

$ curl -u admin:secret -s http://test.app.com/doSomething -o /dev/null -w "%{http_code} %{url_effective}\\n"
200 http://test.app.com/doSomething

I have a list of hosts and paths I want to check regularily using a GET request and the output should show me the response status and the url that was called. So, for a set of hosts tests.app.com, a1.app.com, p.app.com and the paths "/doSomething" and "doSomethingElse" I would run the script and expect the following output

200 http://test.app.com/doSomething
200 http://test.app.com/doSomethingElse
200 http://a1.app.com/doSomething
404 http://a1.app.com/doSomethingElse
200 http://p.app.com/doSomething
500 http://p.app.com/doSomethingElse

This way I can see that the path doSomethingElse does not work.

Now my script looks like this:

#!/bin/bash

HOSTS=(
        'test.app.com'
        'a1.app.com'
        'p.app.com'
)
PATHS=(
        'doSomething'
        'doSomethingElse'
)

USER=admin
PASS=secret

# fixed parameters
auth_header="-u ${USER}:${PASS}"
out_parms="-w \"%{http_code} %{url_effective}\\\n\""

for h in "${HOSTS[@]}"; do
        for p in "${PATHS[@]}"; do
                curl ${auth_header} -s http://${h}/${p} -o /dev/null ${out_parms}
        done
done

It produces several curl calls (similar to the example at the top), but when I run this script, the output looks like the following

"200"000"200"000"200"000"200"000"200"000"200"000

Note: The "200" seems to be the status code and there are 6 requests in my setup.

If I add an echo before the curl command (echo curl ${auth_header} ...), I get a list of commands that I can execute from the command line and which lead to the expected result.

What am I doing wrong?

Upvotes: 1

Views: 1242

Answers (1)

Matt Slonetsky
Matt Slonetsky

Reputation: 306

Getting bash to print out what it's executing for your curl command with 

(set -x; curl ${auth_header} -s http://${h}/${p} -o /dev/null ${out_parms})

the following output is generated:

++ curl -u admin:secret -s http://test.app.com/doSomething -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000++ curl -u admin:secret -s http://test.app.com/doSomethingElse -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000++ curl -u admin:secret -s http://a1.app.com/doSomething -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000++ curl -u admin:secret -s http://a1.app.com/doSomethingElse -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000++ curl -u admin:secret -s http://p.app.com/doSomething -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000++ curl -u admin:secret -s http://p.app.com/doSomethingElse -o /dev/null -w '"%{http_code}' '%{url_effective}\\n"'
"301"000

Looking at 

-w '"%{http_code}' '%{url_effective}\\n"'

We can see that the curl command isn't being formatted the way you'd like

As answered here, you should always put double quotes around variable substitutions; though, there's no reason to parameterize out_parms. Putting that definition inline like this:

curl ${auth_header} -s http://${h}/${p} -o /dev/null -w "%{http_code} %{url_effective}\n"

gave me this output in the format you'd expected.

301 http://test.app.com/doSomething
301 http://test.app.com/doSomethingElse
301 http://a1.app.com/doSomething
301 http://a1.app.com/doSomethingElse
301 http://p.app.com/doSomething
301 http://p.app.com/doSomethingElse

Alternatively, if you want to keep out_parms, the same output is generated if you change out_parms to

out_parms="-w %{http_code} %{url_effective}\n"

and put double quotes around it in the curl command

curl ${auth_header} -s http://${h}/${p} -o /dev/null "${out_parms}"

Upvotes: 3

Related Questions