What kinds of out-of-band failures am I forgetting to test?

Question

I came across an amazing presentation years ago (which of course I can't find) that listed a bunch of kinds of failures for remote services that people usually don't test for.

In addition to timeout, 4xx, 5xx, etc, it listed things like:

• connection closes after 10 bytes of data
• returns contents of www.google.com
• returns contents of /dev/random
• returns contents of /etc/passwd
• returns correctly-formatted unicode chinese text
• returns ansi color control characters
• returns an incorrect content-type, labeled correctly (You requested Content-Type: application/json, I send back Content-Type: application/jpeg)
• returns one byte of data every 29 seconds

What are some types of "out-of-band failures" you've encountered that developers don't usually (but should) test for?

(extra bonus points if you can find the original presentation)

Answer 1

Low-frequency failures. In other words, test that some response is correct not just once, but every time out of a thousand tries. You'll get random Internet breakage if you're going over a network, but you might expose some process is stochastic when you thought it was fixed.

Answer 2

The ones you listed are great; I'd love to see the original presentation if you dig it up! A couple other favorites:

• A "valid" response with a couple bits flipped
• A "valid" response with extra data you weren't expecting ({"result": 123, "extraStuff": {...}}) to simulate upgrades to the remote side
• A syntactically-valid response that never ends ({"results":["lol", "lol", "lol", ..., or just a bunch of whitespace)