iOS Heart rate detection Algorithm

Question

I'm trying to implement heart beat recording functionality in an app i'm developing.

The preferred method of doing this is by using the iPhone's camera with the light on, having the user place their finger on the lens, and detecting fluctuations in the video feed, which correspond to the user's heart.

I found a very good starting point with the following stack overflow question here

The question provides useful code to plot a heart beat time graph.

It shows how to start an AVCaptureSession and turn the camera's light on like so:

session = [[AVCaptureSession alloc] init];

AVCaptureDevice* camera = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
if([camera isTorchModeSupported:AVCaptureTorchModeOn]) {
    [camera lockForConfiguration:nil];
    camera.torchMode=AVCaptureTorchModeOn;
    //  camera.exposureMode=AVCaptureExposureModeLocked;
    [camera unlockForConfiguration];
}
// Create a AVCaptureInput with the camera device
NSError *error=nil;
AVCaptureInput* cameraInput = [[AVCaptureDeviceInput alloc] initWithDevice:camera error:&error];
if (cameraInput == nil) {
    NSLog(@"Error to create camera capture:%@",error);
}

// Set the output
AVCaptureVideoDataOutput* videoOutput = [[AVCaptureVideoDataOutput alloc] init];

// create a queue to run the capture on
dispatch_queue_t captureQueue=dispatch_queue_create("catpureQueue", NULL);

// setup our delegate
[videoOutput setSampleBufferDelegate:self queue:captureQueue];

// configure the pixel format
videoOutput.videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey,
                             nil];
videoOutput.minFrameDuration=CMTimeMake(1, 10);

// and the size of the frames we want
[session setSessionPreset:AVCaptureSessionPresetLow];

// Add the input and output
[session addInput:cameraInput];
[session addOutput:videoOutput];

// Start the session
[session startRunning];

Self in this example must be an <AVCaptureVideoDataOutputSampleBufferDelegate> And will therefore have to implement the following method to obtain raw camera data:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
static int count=0;
count++;
// only run if we're not already processing an image
// this is the image buffer
CVImageBufferRef cvimgRef = CMSampleBufferGetImageBuffer(sampleBuffer);
// Lock the image buffer
CVPixelBufferLockBaseAddress(cvimgRef,0);
// access the data
int width=CVPixelBufferGetWidth(cvimgRef);
int height=CVPixelBufferGetHeight(cvimgRef);
// get the raw image bytes
uint8_t *buf=(uint8_t *) CVPixelBufferGetBaseAddress(cvimgRef);
size_t bprow=CVPixelBufferGetBytesPerRow(cvimgRef);
float r=0,g=0,b=0;
for(int y=0; y<height; y++) {
    for(int x=0; x<width*4; x+=4) {
        b+=buf[x];
        g+=buf[x+1];
        r+=buf[x+2];
        //          a+=buf[x+3];
    }
    buf+=bprow;
}
r/=255*(float) (width*height);
g/=255*(float) (width*height);
b/=255*(float) (width*height);

float h,s,v;

RGBtoHSV(r, g, b, &h, &s, &v);

// simple highpass and lowpass filter 

static float lastH=0;
float highPassValue=h-lastH;
lastH=h;
float lastHighPassValue=0;
float lowPassValue=(lastHighPassValue+highPassValue)/2;

lastHighPassValue=highPassValue;

    //low pass value can now be used for basic heart beat detection


}

RGB is converted to HSV and it is Hue that is monitored for fluctuations.

And RGB to HSV is implemented as follows

void RGBtoHSV( float r, float g, float b, float *h, float *s, float *v ) {
float min, max, delta; 
min = MIN( r, MIN(g, b )); 
max = MAX( r, MAX(g, b )); 
*v = max;
delta = max - min; 
if( max != 0 )
    *s = delta / max;
else {
    // r = g = b = 0 
    *s = 0; 
    *h = -1; 
    return;
}
if( r == max )
    *h = ( g - b ) / delta; 
else if( g == max )
    *h=2+(b-r)/delta;
else 
    *h=4+(r-g)/delta; 
*h *= 60;
if( *h < 0 ) 
    *h += 360;
}

The low pass value calculated in capureOutput: initially provides erratic data, but then stabilises to the following:

2013-11-04 16:18:13.619 SampleHeartRateApp[1743:1803] -0.071218
2013-11-04 16:18:13.719 SampleHeartRateApp[1743:1803] -0.050072
2013-11-04 16:18:13.819 SampleHeartRateApp[1743:1803] -0.011375
2013-11-04 16:18:13.918 SampleHeartRateApp[1743:1803] 0.018456
2013-11-04 16:18:14.019 SampleHeartRateApp[1743:1803] 0.059024
2013-11-04 16:18:14.118 SampleHeartRateApp[1743:1803] 0.052198
2013-11-04 16:18:14.219 SampleHeartRateApp[1743:1803] 0.078189
2013-11-04 16:18:14.318 SampleHeartRateApp[1743:1803] 0.046035
2013-11-04 16:18:14.419 SampleHeartRateApp[1743:1803] -0.113153
2013-11-04 16:18:14.519 SampleHeartRateApp[1743:1803] -0.079792
2013-11-04 16:18:14.618 SampleHeartRateApp[1743:1803] -0.027654
2013-11-04 16:18:14.719 SampleHeartRateApp[1743:1803] -0.017288

An example of the erratic data provided initially is here:

2013-11-04 16:17:28.747 SampleHeartRateApp[1743:3707] 17.271435
2013-11-04 16:17:28.822 SampleHeartRateApp[1743:1803] -0.049067
2013-11-04 16:17:28.922 SampleHeartRateApp[1743:1803] -6.524201
2013-11-04 16:17:29.022 SampleHeartRateApp[1743:1803] -0.766260
2013-11-04 16:17:29.137 SampleHeartRateApp[1743:3707] 9.956407
2013-11-04 16:17:29.221 SampleHeartRateApp[1743:1803] 0.076244
2013-11-04 16:17:29.321 SampleHeartRateApp[1743:1803] -1.049292
2013-11-04 16:17:29.422 SampleHeartRateApp[1743:1803] 0.088634
2013-11-04 16:17:29.522 SampleHeartRateApp[1743:1803] -1.035559
2013-11-04 16:17:29.621 SampleHeartRateApp[1743:1803] 0.019196
2013-11-04 16:17:29.719 SampleHeartRateApp[1743:1803] -1.027754
2013-11-04 16:17:29.821 SampleHeartRateApp[1743:1803] 0.045803
2013-11-04 16:17:29.922 SampleHeartRateApp[1743:1803] -0.857693
2013-11-04 16:17:30.021 SampleHeartRateApp[1743:1803] 0.061945
2013-11-04 16:17:30.143 SampleHeartRateApp[1743:1803] -0.701269

The low pass value goes positive whenever there is a heart beat. So I tried a very simple live detection algorithm which basically looks at the current value, and sees if it is positive, it also looks at the previous value, if negative it detects negative going to positive and plays a beep sound.

The problem with this is the data isn't always as perfect as the above, sometimes there's anomalous positive readings in amongst negative readings and vice versa.

A graph of the low pass value in time looks like this:

Interestingly the above anomaly is quite common, if I record a graph for a while i'll see a very similar shaped anomaly multiple times.

In my very simple beat detection algorithm, if an anomaly as shown above occurs the counted number of beats in the detection period (10 seconds) can shoot up by 4 or 5 beats. This makes the calculated BPM very inaccurate. But as simple as it is it does work around 70% of the time.

To combat this problem I tried the following.

1.Started recording last 3 low pass values in an array

2.Then looked to see whether or not the middle value had two smaller values surrounding it before and after. (Basic peak detection)

3.Counted this scenario as a beat and added it to the running total of beats in a given time.

This method is however just as vulnerable to the anomalies as any other. And actually seemed to be a worse method. (When playing live beeps after detection they seemed far more erratic than the positive to negative algorithm)

My question is can you help me come up with an algorithm that can reliably detect when a heart beat occurs with reasonable accuracy.

Another problem I realise that i'm going to have to address is detecting whether or not a user's finger is on the lens.

I thought about detecting erratic low pass values but the problem there is the low pass filter accounts for erratic values and smooths them out over time. So help there would be appreciated too.

Thanks for your time.

Answer 1

I made a project that uses GPUImage filters, average color and exposure, to detect your pulse. The pulse is estimated based the running average of the green component of the filtered image.

Optical Pulse Reader

Answer 2

The answer to this question is a little bit involved, as you need to do several things to process the signal and there is no single "right" way to do this. However, for your filter you want to use a band-pass filter. This type of filter allows you to specify a range of frequencies that are accepted both at the high and low ends. For a human heart beat, we know what these bounds should be (no less than 40 bpm and no higher than 250 bpm) so we can create a filter that removes frequencies outside of this range. The filter also moves the data to be centered at zero, so peak detection becomes much easier. This filter will give you a much more smooth signal, even if your users increases/decreases their finger pressure (to a certain degree). After that, additional smoothing and outlier removal will also need to happen.

A specific type of band-pass filter that I have used is a butterworth filter. This is a little involved to create by hand since the filter changes based on the frequency you are collecting your data at. Fortunately, there is a website that can help with this here. If you are collecting your data at 30 fps, then the frequency will be 30 hz.

I have created a project that wraps all of this together and detects a user's heart rate well enough to include it in my app on the iOS app store. I have made the heart rate detection code available on github.

Answer 3

I would :

1) Detect the period from peak to peak... If the period is consistent within a certain period threshold.. Then flag the HB as valid.

2) I would implement an outliar detection algorithm... Any beat that goes beyond a certain normalized threshold.. Would be considered an outlier an thus i would use the last detected beat instead to compute my BPM.

Another more complex approach would be to use a Kalman filter or something of sorts to be able to predict the bpm and get more accurate readings.. It is only after a few skips that the app will sense that the readings are not valid and stop the reading.

Answer 4

First addressing your finger-on-the-lens problem. When the finger is on the lens, you don't get a static black frame (as one might assume). The ambient light actually passes through your finger to create a reddish frame. Also, the blood flow pattern in the finger results in slight periodic variations in that red frame (try opening your camera app, placing your finger completely on the lens). Also, if the ambient light isn't enough, you can always turn on the camera flash/torch to compensate for that.

For an open source (and step by step) procedure, try:

http://www.ignaciomellado.es/blog/Measuring-heart-rate-with-a-smartphone-camera

also, I would advise you to read the following patent on pulse measurement:

http://www.google.com/patents/WO2013042070A1?cl=en

Answer 5

I presume you're using your own finger. Are you sure you don't have an irregular heartbeat? Plus, you're going to want to handle people with irregular heartbeats. In other words, you should test with a wide variety of input values. Definitely try it on your parents or other older relatives, as they might be more likely to have heart issues. Other than that, your basic problem is that your input source is going to be noisy; you're basically trying to recover signal from that noise. Sometimes that will be impossible, and you're going to have to decide if you want to bake noise into your report or just ignore the data stream when it's too noisy.

Keep trying different filter values; maybe you need an even lower pass filter. From the comments, it sounds like your low pass filter was not good; there's tons of resources on filtering out there in the web. If you've got good visualization tools that will be the best way to test your algorithm.

You can try down-sampling the data, which will smooth it out. You might also want to discard samples which lie outside a valid range, either by discarding the value altogether, replacing it with the average of the previous and next sample, and/or by clamping it to some predetermined or calculated maximum.

My biggest beef with these sorts of applications is that hiccups are treated as real, live data. One of the bikes at my gym gives useless bpm readings because, every so often, it can't find my pulse and suddenly thinks my heart's going at 300 bpm. (Which it isn't; I've asked my doctor.) For a 20-minute session the average it has is useless. I think it's more useful to see the average of the (e.g.) last ten normal beats plus the anomaly rate rather than "I tried to cram the last 20 seconds into this algorithm and here's the garbage it spat out". If you can create a separate filter that indicates anomalies, I think you'll have a much more useful application.

Answer 6

You're trying to do detect a single heart beat "manually", that won't be very robust. I'd say that your best bet is something like a pitch or frequency detection library (the math for detecting the frequency of a color change and for detecting the frequency of a sound has to be identical).

Perhaps something like aubio which I found via this stackoverflow answer to the search "frequency detection" can help you. Otherwise check wikipedia for pitch detection and/or some of the tons of signal processing libraries out there.

Answer 7

It sounds like you may already have another method, but one thing you could try is to use a small median filter. For example, using the median of, say, 3 to 7 input values for each output value will smooth out those peaks without destroying the overall shape of the non-anomylous data.