CODEWITHSUNDEEP
bashcurl
Adrian
Adrian

Reputation: 2656

Count pattern in curl output for a list of URLs

How it is possible to count a number of times a pattern occurs in curl output for a list of URLs, in bash?

For example, if URLs in url.txt are:

xy.com/test.php 
xy.com/test2.php
xy.com/test3.php

and both test.php and test2.php return:

{"error":null,"result":true}

and I want to count how many times a response {"error":null,"result":true} occurs, when I execute a command, to get this:

$ curl -i url.txt ....
...
...
Matched result: 2

Upvotes: 1

Views: 2399

Answers (2)

Ulises André Fierro
Ulises André Fierro

Reputation: 602

With command substitution, it would look like this:

curl -i $(cat url.txt)|grep -c '{"error":null,"result":true}'

Piping the output to grep -c makes it so that grep will return the number times the string you're looking for was matched.

Upvotes: 1

randomir
randomir

Reputation: 18687

If I'm understanding your question correctly, you wish to fetch all URLs from the url.txt file (one per line) with curl and count the number of times the {"error":null,"result":true} string appears.

You can do it like this:

$ <url.txt xargs -n1 curl -s -i | grep -F '{"error":null,"result":true}' -c
2

We pipe the url.txt to xargs (assuming URLs are properly quoted, without whitespaces) which calls curl with one URL at a time (due to -n1 option). Progress output from curl is silenced with -s, and -c tells grep to output the count instead of matches, while -F looks only for fixed string given (not pattern matches).

Upvotes: 3

Related Questions