CODEWITHSUNDEEP
bashcurl
Jonathan Mann
Jonathan Mann

Reputation: 23

How to check the validity of a list of URLs in a file?

Trying curl a list of urls to confirm they are active or not. I'd like to separate active from non-active but I'm stuck. I'm super new to bash scripting, as I'm sure you can tell. cat function into the url doesn't appear to work correctly. Separating the contents also doesn't work correctly. A large potion of the urls will go into nv.txt but I know they are active urls.

Thank you for the help in advance

#!/bin/bash
url=$(cat non-ssl-list2.txt)
if curl --output /dev/null --silent --head --fail "$url"
then
    echo $url Exists > v.txt
else
    echo "$url Does Not Exist" > nv.txt
fi

https://hastebin.com/tuqekocate.bash

(code kept getting chopped off so I put it into hastebin).

https://hastebin.com/nogasiyeyo.css (sample of list)

Upvotes: 2

Views: 2070

Answers (2)

codeforester
codeforester

Reputation: 43039

The issue in your code is here:

url=$(cat non-ssl-list2.txt)

this slurps all lines of your input file into a single string. That's not what you want, right?

You can loop through the urls in your file line by line and invoke curl to test if the URL is valid:

while read -r url; do                    # loop through the array
  if curl -s --head "$url" > /dev/null; then  # check URL validity
    printf '%s\n' "$url exists"
  else
    printf '%s\n' "$url does not exist"
  fi
done < non-ssl-list2.txt

See these related posts:

Upvotes: 1

iamauser
iamauser

Reputation: 11479

One can use && and || operators to print to an output file based on the exit status of the curl command.

$ curl -k -s -o /dev/null "${url}" && echo "${url} : Exists" > 1.txt || echo "${url} : Does not exist" > 2.txt

For a list,

#!/bin/bash
> 1.txt  # Create empty text file
> 2.txt  # 
while IFS= read -r url
do
  curl -k -s -o /dev/null "${url}" && echo "${url} : Exists" >> 1.txt || echo "${url}: Does not exist" >> 2.txt
done < /path/to/list.txt

Upvotes: 1

Related Questions