CODEWITHSUNDEEP
linuxbashurlcurlsed
Danish Sattar
Danish Sattar

Reputation: 89

CURL on list of links to save only specific links along with specific response

Hi All i have list of links, through curl I am getting their response by using this command curl -K file.txt.

I am saving response to file.

But the problem is I am getting rough response(collectively all curl request result in one file), I want to filter the response in two possible ways.

1 - End result in file should be organized like this URL against each response or maybe if other possible organized way. e.g.

Heading http://192.168.0.1

Response (includes all responded html)

2 - Once first part is done, i want to keep only those urls and response which have specfic keywords. e.g i want only "helloworld" in urls response to stay in final result file. if there is no particular keyword in response against any link from list remove those links and their result from file.

list of urls

Upvotes: 0

Views: 185

Answers (1)

Jay jargot
Jay jargot

Reputation: 2868

This example is assuming that file.txt exist.

It will test the existence of HelloWorld in the response.

The If statements had been written on 3 lines for readability.

The responses.txt will contain the result.

An intermediary file is used to store each http response in turn: /var/tmp/response-tmp.txt

Give a try to this:

regex="HelloWorld"
rm -f responses.txt 2>/dev/null
for url in $(sed -n 's/^[[:blank:]]*url[[:blank:]]*=[[:blank:]]*//gIp' file.txt); do
  curl -w "http_code %{http_code}" -Ns --url "${url}" > /var/tmp/response-tmp.txt 2>/dev/null
  if [[ $? -eq 0 ]] ; then
    if tail -1 /var/tmp/response-tmp.txt 2>/dev/null | grep -aq "http_code 2[0-9][0-9]$" 2>/dev/null ; then 
      if grep -aq "${regex}" /var/tmp/response-tmp.txt 2>/dev/null ; then
        printf "Heading %s\nResponse " "${url}"
        head -c -13 response-tmp.txt
      fi
    fi
  fi
done >> responses.txt

Upvotes: 1

Related Questions