How to write this integration test

Question

I am trying to help my uncle out who is in a bit of a jam. An application he had a consultant come in on has stopped working and the consultant has moved on from the company and no one can fix the issue. He's asked me to try and help by isolating where it is going wrong.

I figure writing an integration test would be a good start as we would be programmatically testing on their 'test' instance. Here's the setup:

They're pulling in streams of data every 10 minutes and sending these orders to an AMQ instance they have running. (we'll call this module 'Streamer'). On the other side of the AMQ instance we have the 'Puller' that takes those messages off AMQ and processes them into a database. Somewhere along the line the messages are being manipulated or lost. He's asked me to find out where.

Does anyone know of a simple way in which I can test the entire system and yet have the ability to verify all the information as it's being sent around or updated? (I would like to do this with live data, rather than mocked up data).

Any and all help is appreciated.

Answer 1

That's not strictly speaking an integration test. An integration test is testing that new code plays well with other code before you let it go live.

What you need is logging. Each thing that handles these messages needs to have it's logging turned on, at least until you're sure it's working.

I've no idea if your consultant added logging code to this thing but if s/he didn't I suggest adding some.

Keep in mind that logging can be very granular. You can focus on one method and even one variable. Start about halfway between where you know it's working and where you know it's broken and find out if it's working there. Repeat until you know exactly where it stopped working. It's called a binary search. It can be a very powerful troubleshooting technique.

Another technique is to send a heartbeat message from your streamer to the puller. You could send a very simple message quite often so you wouldn't need to wait 10 minutes for activity. Just need to be sure the heartbeat message doesn't get confused with real data.

Answer 2

You may need the combination of logging and a post-processing program that reads the logs and checks their consistency.

Streamer sends a message. That should generate a log record describing the message, with a time stamp.

There should be another log record showing that Puller received the message.

A log processing program could produce a histogram of how long it takes for Puller to receive what Streamer sent, and detailed accounting for any messages that Streamer sent and Puller did not receive, or that Puller thought it had received but that had not been sent. You can also extract statistics about any long gaps in activity.

The objective of your logging should be to make sure that the raw data gets into the logs. The objective of the log processing should be to tell you what you need to know about the system's activity, at the right level of detail.