programmatically checking if a website is blocked by ISP

Question

I need to monitor a list of websites through different ISPs to see if they are blocked. I have a different machine for each ISP. I'm trying to write a code to automatically check if the websites are blocked. A few things came to my mind but they are not working for different reasons:

ping: I thought I would ping websites but then some websites have their ICMP ports closed on their side.

get request (or javascript image trick): I thought maybe I just GET the webpage but that wouldn't help because the blocked pages still return some non-standard blockage page. 200 status.

Lastly, I thought maybe I get a copy of the website on a non-blocked machine and compare the page with the one on testing machines but there are 2 problems: I don't know how to compare two pages (like what element would I compare) and secondly, some websites are dynamic thus they return slightly different versions.

Any thoughts will be helpful.

Answer 1

store the template of the blocked site page.

For each GET responseBody check if its contents match to blocked template.

If the firewall forces a redirect you could check if the IP/DNS of the response server is equal to the firewall.

Answer 2

Two Options:

Compare the size of the page. If two page's sizes are fairly similar then you know the ISPs are likely not blocking the site. If one page's size is extremely small, chances are that ISP is blocking the site.

Grab elements from the pages such as headers, titles, button text, etc. and compare those to each other. If you have x matches the page is likely not blocked, if you have no matches the page is likely blocked.

Answer 3

I'm guessing that the ISP block page is consistent to that ISP no matter what site you're trying to access. Instead of comparing the retrieved page to a "known true" page, what about comparing it to a "known false" page?

Downside: you would have to repeat this process for each ISP, since they will almost certainly have different "block" pages.