Ways to programatically check if a website is up and functioning as expected

Question

I know this is an open ended question, but hopefully it will get some good answers before the thread is locked...

I'm wondering what methods there are to programmatically check (language agnostic) if a website is online from a client perspective (assume you can't make changes to the site/server, but you can rely on certain behaviours of the site.)

The result of each method could stack to provide a measure of certainty that the site is up/down - that is, a method does not have to provide a definite indication if the site is up/down on its own.

Some common tests just to check 'upness' may be:

• Ping the site (which in the case of shared hosting isn't very indicative)
• Send a http head/get request and check the status

Others I can think of to check that the site is up and functioning:

• Check you received a well formed html response i.e. html to html tags, if the site is experiencing trouble it may spit an error and exit without writing the rest of the page (not all that reliable though because the site may handle most errors in a better way)

• Check certain content is or is not on the page, i.e. perhaps there is some content that is always present on your pages, or always present in the case of an error

Can anybody think of any other methods that could be used to help determine if a site is in fact up/down and functioning/not functioning correctly from within a program?

Answer 1

You can deploy your own service to do this, clone https://github.com/DALIHILLARY/ping and set and monitor at your convenience. The system will check all your domain/ websites every five minutes and will send you email notifications when something goes wrong with your websites

Answer 2

If your get request on a page that displays info from database comes back with status 200 and matching keywords are found, you can be pretty certain that your site is up and running.

And you don't really need to write your own script to do that. There are free services such as GotSiteMonitor, Pingdom, UptimeRobot etc. allows you to monitor your site.

Answer 3

OK, it sounds like you want to test and monitor your website from a customer experience perspective rather than purely establishing if a server is up (using ping for example). An effective way to replicate the customer experience is to simulate tests against the site using one of the headless browser testing tools (phantomJS is great a great choice) as they will render the page fully (including images, CSS, JS etc.) giving you a real page load time. These tools also allow you to make assertions on all aspects of the HTML content and HTTP response.

pingdom recently started offering a (paid for) service to perform these exact types of checks for alongside their existing monitoring solution. The demo is worth looking at, their interface for writing the actual tests is very nice.

Answer 4

Based your set of test on the unit tests priciple. It is normally used in programming to test classes, modules or other artefacts after changes have been made. You can use any of the available frameworks, so don't have to reinvent the wheel. You must describe (implement) tests to be run, in your case a typical test should request a url inside the page and then do some evaluations like:

• call result (for example return code of curl execution)
• http return code
• http headers
• response mime type
• response size
• response content (test against a regular expression)

This way you can add, remove and modify single tests without having to care about the framework, once you are up. You can also chain tests, so perform a login in one test and virtually click a button in subsequent test.

There are also tools to handle such test runs automatically including visualization of results, statistics and the like.