CODEWITHSUNDEEP
bash
Edmar
Edmar

Reputation: 37

Getting a piped curl command output to a variable in bash

I'm attempting to monitor an output from a proxy server to troubleshoot issues, in order to do that I've created a bash script which uses curl, but I'm having trouble in getting the output in a variable.

I've tried different ways to escape and to use the variables, either using ticks or parentheses.

I was using this first, but wasn't working properly:

response=$(curl -k -x $i:1234 -U $u:$p -w $outputformat -s -o /dev/null $site | cut -d ' ' -f 2)

Thus changed it, and the code now looks like this:

response=$(curl\ -k\ -x\ $i:1234\ -U\ $u:$p\ -w\ \"HTTP\ Response:\ \%\{http\_code\}\ \-\ URL:\ \%\{redirect\_url\}\\n\"\ -s\ -o\ /dev/null\ $site\ \|\ cut\ -d\ \'\ \'\ -f\ 2)
if [ "$response" -eq 200 ]; then
   ((count200++))
   echo $count200 > $counter_200
else
   ((countx0x++))
   echo $count0x0 > $counter_x0x
fi

8/22 - Updated code as requested, I reverted back the code to what it was before and also following WaltDe's recommendation removed the -o option:

echo "Proxy: $i"
echo "Fetching site: $site"
outputformat="\"HTTP Response: %{http_code} - URL: %{redirect_url}\n\"" 
response=$(curl -k -x $i:$port -U $u:$p -w $outputformat -s $site | cut -d ' ' -f 2)
if [ "$response" -eq 200 ]; then
   ((count200++))
   echo $count200 > $counter_200
else
   ((countx0x++))
   echo $count0x0 > $counter_x0x
fi

This is the error message that I'm seeing:

./ptest.sh: line 80: curl -k -x *********:** -U : -w "HTTP Response: %{http_code} - URL: %{redirect_url}\n" -s -o /dev/null https://www.yahoo.com | cut -d ' ' -f 2: No such file or directory

./ptest.sh: line 82: [: : integer expression expected

8/22 Update - This is the issue I'm having now:

Proxy: ************ Fetching site: https://www.yahoo.com html> id="atomic" (...) shortened for readability - html output (...) Response:: integer expression expected ./ptest.sh: line 83: [: Response:: integer expression expected

Could anyone shed some light as to what I'm doing wrong?

Upvotes: 1

Views: 1835

Answers (1)

WaltDe
WaltDe

Reputation: 1832

You need to remove the -o option that writes the output to a file, in your case /dev/null. Here is your updated code that should work. 

$ response=$(curl -k -x $i:1234 -U $u:$p -w $outputformat -s $site | cut -d ' ' -f 2)

Another way to check is to remove pipe and send your STDERR to /dev/null. This will show you what is getting passed to your cut command.

You should see something if all is happy.

$ curl -k -x $i:1234 -U $u:$p  -s $site 2>/dev/null

If you don't get any output, then remove the STDERR redirect and echo the return code. 

$ curl -k -x $i:1234 -U $u:$p  -s $site
$ echo $?
0

If the exit code is anything other then 0 you have an error. You can find the error codes in the man page or this site has pretty good write up.

No to get the response code.

$ curl  -w "HTTP Response: %{http_code} - URL: %{redirect_url}\n"  -o /dev/null  -s http://yahoo.com/
HTTP Response: 301 - URL: https://yahoo.com/
$  curl  -w "%{http_code}\n"  -o /dev/null  -s http://yahoo.com/
301
$ response=$(curl  -w "%{http_code}\n"  -o /dev/null  -s http://yahoo.com/)
$ echo $response
301
$  curl  -w "%{http_code}\n"  -o /dev/null  -s http://yahoo.com/ | cut -d' ' -f3
301
$ response=$( curl  -w "%{http_code}\n"  -o /dev/null  -s http://yahoo.com/ | cut -d' ' -f3)
$ echo $response
301

Upvotes: 2

Related Questions