CODEWITHSUNDEEP
c#.net.net-corehttpwebrequest
Rogier
Rogier

Reputation: 915

Check is url is valid by using HttpWebRequest

I'd like to create a tool to check if an url is valid (valid: it returns a 200). I have two examples of check in pages of airlines, and both works correctly in the browser. However the British Airlines always throws an exception becuase of a 500 response. What is wrong with my code?

static void Main(string[] args)
    {
        var testUrl1 = new Program().UrlIsValid("https://www.klm.com/ams/checkin/web/kl/nl/nl");
        var testUrl2 = new Program().UrlIsValid("https://www.britishairways.com/travel/olcilandingpageauthreq/public/en_gb");
        Console.WriteLine(testUrl1 + "\t - https://www.klm.com/ams/checkin/web/kl/nl/nl");
        Console.WriteLine(testUrl2 + "\t - https://www.britishairways.com/travel/olcilandingpageauthreq/public/en_gb");
    }

    public bool UrlIsValid(string onlineCheckInUrl)
    {
        try
        {
            var request = (HttpWebRequest)WebRequest.Create(onlineCheckInUrl);
            request.Method = "GET";
            var response = (HttpWebResponse)request.GetResponse();

            return (response.StatusCode == HttpStatusCode.OK);
        }
        catch (Exception e)
        {
            return false;
        }
    }

Upvotes: 0

Views: 570

Answers (1)

Crowcoder
Crowcoder

Reputation: 11514

A lot of sites block obvious bot activity. The British Airways url you show works for me if I set a valid User-Agent request header:

request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:87.0) Gecko/20100101 Firefox/87.0";

Keep in mind that 200 OK is not the only response that means the URL is valid and your method of testing will always be unreliable at best. You may have to narrow your definition of what a valid URL means or at least expect things to change on a site-by-site basis.

Upvotes: 1

Related Questions