Finding all points on a slope of a signal

Question

I have a 1d signal that is given as following

x = np.array([34.69936612, 34.70083619, 37.38802174, 39.67141565, 49.05662135,
       63.87593075, 67.70815746, 72.06562117, 79.31063707, 85.13125285,
       83.34185985, 72.74589905, 57.34778159, 58.63283664, 64.92526896,
       65.89153823, 66.07273386, 59.68722257, 59.6801125 , 59.41456929,
       58.19250575, 59.92192524, 58.42078866, 55.45131784, 55.09849914,
       54.95270916, 49.60804717, 43.05198366, 36.10104167, 26.88848229,
       25.38550393, 28.71305461, 30.03802157, 31.3520023 , 32.59509437,
       32.67600055, 32.68801666, 32.61500098, 32.65303828, 32.72752018,
       32.84099458, 31.46154937, 32.70809456, 27.67842221, 25.65302641,
       30.08500957, 31.41003082, 32.91935844, 32.92452782, 35.56587345,
       30.09272452, 35.60898454, 49.12005244, 85.79396522, 71.81950127,
       63.91915245, 69.14879246, 70.43600086, 71.71703424, 71.74830965,
       70.51400086, 70.50201501, 70.50202228, 67.91157904, 66.62396413,
       67.90736076, 66.5410636 , 67.96748026, 67.94177515, 65.30929726,
       65.29901863, 66.60282538, 66.60666811, 66.55100589, 65.33825435,
       66.55222626, 65.29656691, 66.56003543, 65.30964145, 64.07556963,
       63.99339626, 62.86668124, 60.43549001, 61.68116229, 61.61140279,
       62.65181523, 62.70844205, 62.77783077, 64.03882299, 65.39701193,
       65.40123835, 65.41845477, 65.42941287, 65.38851043, 65.36201151,
       73.33102635, 73.84443755, 70.94806114, 68.18793023, 69.20003749,
       66.61045573, 65.38106858, 64.05484531, 63.88684974, 62.64420529,
       62.69196131, 62.74418993, 62.72175294, 64.01210311, 64.1590297 ,
       63.0284751 , 64.27265024, 64.24984689, 62.90213438, 62.68704697,
       62.65233151, 63.09040365, 63.10330994, 62.72787413, 63.95427977,
       63.89707325, 61.38203635, 57.48587612, 60.05764178, 62.70293674,
       61.38484666, 60.07995823, 61.34569129, 62.66307354, 61.38549663,
       61.34835356, 61.3888718 , 61.48381576, 62.74226583, 62.83945058,
       41.78731982, 38.06452548, 40.57553545, 43.10410628, 43.17965777,
       44.41576623, 45.67422069, 44.44681128, 44.52855717, 45.69118569,
       45.7559632 , 49.9019806 , 50.90898633, 52.2603325 , 36.83061979,
       48.36714502, 53.60110239, 53.58750501, 51.03745637, 52.15201941,
       50.94600264, 48.50758345, 51.03154956, 51.32249134, 51.49705585,
       53.46467209, 51.708078  , 48.1404585 , 46.32157084, 53.20416229,
       60.52216104, 67.14976382, 66.6844348 , 63.99400013, 63.89292312,
       63.94972283, 65.33551293, 66.54723199, 65.29004129, 67.87224117,
       69.3810433 , 69.28915977, 65.32064534, 64.07644938, 64.59988251,
       65.55365125, 64.3440046 , 64.4526091 , 63.38977665, 64.61810574,
       63.52989024, 63.55126155, 64.4263114 , 64.43874937, 64.78594756,
       66.03974204, 67.34958445, 70.07248445, 67.40968741, 66.56554542,
       67.59965865, 67.85658168, 67.62022101, 67.87089721, 61.22552792,
       54.07823817, 47.96332512, 53.22944931, 54.77573267, 59.55033053,
       62.24247612, 62.24529416, 63.9429676 , 63.13145527, 63.29764489,
       63.2723988 , 62.96359318, 63.3025575 , 63.47790181, 63.29642863,
       63.50702402, 63.71413853, 63.71470992, 62.25079434, 63.46787461,
       63.73497156, 63.77631175, 63.69024723, 63.55254533, 63.97794376,
       64.05815662, 63.57687055, 66.80917018, 66.82863683, 66.27964922,
       65.04852024, 65.29135318, 65.57783886, 65.52090561, 65.29656225,
       65.32543578, 66.52825603, 67.1314033 , 50.03567181, 53.53803024,
       53.56862071, 55.10515723, 55.14010716, 63.30760687, 62.7114906 ,
       62.95237442, 62.75869066, 64.19585539, 62.70371169, 62.65204241,
       62.69394807, 62.94844878, 58.36397143, 59.68285611, 60.89452752,
       60.97356663, 60.72068974, 59.62036073, 60.52789377, 59.27245489,
       58.82200393, 60.10430588, 60.90874661, 61.51060014, 61.74838059,
       63.28503148, 61.12237542, 60.87046418, 61.23634728, 60.99214796,
       60.18921274, 60.07774571, 61.20623845, 61.65825197, 60.11025633,
       60.52832382, 61.18188688, 61.31380433, 58.80528487, 57.84584698,
       58.73805752, 54.85645345, 58.79988199, 60.07737149, 56.20096342,
       60.3929374 , 36.77761826, 49.22568866, 55.10930206, 65.24736292,
       57.08641006, 54.08806036, 53.89556268, 53.5613321 , 53.51515767,
       52.30442805, 52.24562597, 53.50311397, 53.49561038, 53.53878528,
       49.66610081, 52.35633014, 55.17584864, 53.945292  , 53.79353353,
       54.8626422 , 54.87102507, 56.14098197, 57.38968051, 55.1146169 ,
       54.92290752, 54.87858275, 54.86639486, 56.34316676, 56.16200014,
       69.90905494, 68.20948497, 68.51263756, 65.64670149, 65.53992678,
       67.07185321, 67.0542345 , 66.79344433, 66.75400526, 66.76640135,
       66.76742739, 65.53052634, 67.01174217, 67.98329773, 69.18915578,
       66.69019707, 69.61506484, 67.94096632, 67.91401491, 66.84415179,
       67.88935229, 67.89356226, 69.1984958 , 52.24244378, 52.37211419,
       50.95591909, 51.07641848, 50.91919022, 52.13500015, 52.26717303,
       60.03109894, 65.23341727, 72.11099746, 75.02859632, 81.93540828,
       81.20708335, 80.86208705, 81.04817673, 71.74669785, 73.05200134,
       74.34519255, 75.72326992, 78.55812705, 76.95800509, 77.08696036,
       79.61302675, 79.68123466, 78.31207499, 77.08036041, 77.18815309,
       77.11523959, 75.74423094, 75.73143868, 74.48319908, 73.17138546,
       66.80804931, 53.88772644, 53.87714358, 53.6088119 , 53.65411471,
       54.86536613, 53.49300076, 53.52447811, 53.52000034, 56.83649529,
       57.43503283, 82.38440921, 83.83190983, 83.9128805 , 83.94305425,
       83.06892508, 82.91998964, 82.29555463, 82.30635577, 82.23464297,
       82.20709065, 80.98821075, 83.93336979, 81.32873456, 82.46698736,
       82.70592498, 83.93335761, 83.80821766, 83.84313602, 82.59867874,
       82.62361191, 83.94865746, 83.83137976, 83.46075784, 82.14902814,
       82.18902896, 83.83722778, 83.60064452, 83.63187976, 85.04806926,
       84.87213079, 84.92473511, 84.90790341, 83.55500539, 83.59501005,
       84.81195299, 84.86952928, 84.85600059, 84.81955391, 82.33120262,
       78.56908599, 73.14783901, 64.99883861, 66.78701764, 64.5916058 ,
       64.77055337, 64.56918786, 65.02605783, 65.01019955, 64.78145201,
       64.77581828, 64.55221044, 64.34285288, 62.8764752 , 64.57949744,
       63.17957281, 61.89857751, 63.48365778, 55.62801456, 43.17986365])

I want to find all the slopes for this signal. I have tried first order difference and second order difference (np.diff and taking the difference of the difference). But the point on the slope will have every small difference, in contrast to the point in the beginning or the end, where the difference is bigger.

Here is what I have tried

def detect_slope(signal, window_size = 3, threshold = 5):
    list_ = []
    for i in range(window_size, len(signal)-window_size):
        diff_ =  np.mean(signal[i-window_size:i]) - np.mean(signal[i:window_size+i])
        list_.append(diff_)
        
    first_order_diff = np.array(list_)    
    
    d = np.where(np.abs(first_order_diff) < threshold, 0 , first_order_diff)
    idx = np.where(np.abs(d) != 0)
    
    # might need some offset because we are doing some smoothing, but just use raw idx for now
    
    # second order different
    diff_list = np.array(list_).copy()
    dd = np.diff(diff_list)
    print(dd.shape)
    dd_idx = np.where(np.abs(dd) > 0.5)
    
    return diff_list, dd, idx, dd_idx

I have played around the 1st-2nd order difference but nothing seems to work. I'm trying to find all the peaks and troughs and exclude all of them or neighbors with close enough values too.

Attached is my desired output. Sorry for the crappy pic.

Answer 1

You can try find_peaks function from scipy. As you guess it gets the peaks defining a parameter to be more or less sensitive. The best one in your case is prominence ("How much you have to go down before finding another peak"). I use with your function in positive for max peaks and negative for min peaks.

import numpy as np
from matplotlib import pyplot as plt
from scipy.ndimage import median_filter
from scipy.signal import find_peaks
#Find peaks (maximum)
yhat = x
max_peaks,_ = find_peaks(yhat, prominence=10 )
min_peaks,_ = find_peaks(-yhat, prominence=10 )
#Plot data and max peak
fig, ax2 = plt.subplots(figsize=(20, 10))
ax2.plot(max_peaks, yhat[max_peaks], "xr",markersize=20)
ax2.plot(min_peaks, yhat[min_peaks], "xr",markersize=20)
ax2.plot(yhat,'-')
ax2.plot(yhat,'o',markersize=4)

Answer 2

Not clear what you want here, but if your issue is that the diff is picking up local changes and you want to focus your attention on global changes, smooth the signal first.

import numpy as np
import matplotlib.pyplot as plt
from scipy.signal import savgol_filter

plt.plot(x)
x = savgol_filter(x, 21, 3)
plt.plot(x)


diff_list, dd, idx, dd_idx = detect_slope(x)
plt.plot(diff_list)
plt.show()

This gives -

Blue is your original signal, orange is your smoothed signal and green is your new diff. You can set it to pick up changes at various levels by playing around with the two parameters of savgol_filter. The more aggressively you smooth your function, the more global changes(and less local changes) the derivative picks up.