Calculate time shift between two signals and shifting

Question

I have a temperature sensor and a strain sensor, I would like to calculate the time delay between the two sensors.

def process_data_time_delay(temperature, strain, normal_data):
    from scipy import signal

    T1 = temperature['T1'].tolist()
    S1 = strain[0]
    time = normal_data['TIMESTAMP'].tolist()

    fig = plt.figure()
    plt.plot(time, T1, 'r')
    plt.plot(time, S1, 'b')

    shift = (np.argmax(signal.correlate(T1, S1)))
    fig  = plt.figure()
    plt.plot(time, T1 - shift)
    plt.show()

I have a strange and strange output graph.

Here is a sample data

https://pastebin.com/eEd6MTL0

According to the excellent answer I have modified the code to calculate the time delay between the two signals.

def process_data_time_delay(temperature, strain, df):
    from scipy import signal

    # normalization before ACF
    def normalize(data):
        return (data - np.mean(data, axis=0).reshape((1, -11))) / (np.std(data, axis=0).reshape((1, -1)))

    # select subset of columns, seems relevant as a group
    SCOLS = ['T1', 'W_A1']

    # just to see the data
    f = plt.figure()
    ax = f.add_subplot(111)
    df[SCOLS[:2]].iloc[::10].plot(ax=ax)
    ax.set_title('Raw data')

    # normalization
    normalized = normalize(df[SCOLS].values)
    f = plt.figure()
    ax = f.add_subplot(111)
    ax.plot(np.arange(normalized.shape[0]), normalized[:, 0], label='TW_A1')
    ax.plot(np.arange(normalized.shape[0]), normalized[:, 1], label='W_A1')
    ax.set_title('Normalized')

    # ACF between two components
    x1x2 = np.correlate(normalized[:, 0], normalized[:, 1], 'full')

    # see the results
    f = plt.figure()
    ax = f.add_subplot(111)
    ax.plot(x1x2)
    ax.set_title('ACF')
    df['TIMESTAMP'] = pd.to_datetime(df['TIMESTAMP'])
    peaks_indices = signal.find_peaks_cwt(np.array(x1x2), np.arange(1, len(x1x2)))
    print(peaks_indices)
    delta_index = np.argmax(peaks_indices);
    delta_time = df['TIMESTAMP'][delta_index] - df['TIMESTAMP'][0]
    # assuming timestamps is a datetime64 numpy array that can be easily obtained from pandas;
    shifted_signal = x1x2[delta_time:]

    f = plt.figure()
    ax = f.add_subplot(111)
    ax.plot(shifted_signal)

    # mainloop
    plt.show()

    return x1x2

Answer 1

Your question is not about python, it is about signal processing. There are many techniques and issues around phase estimation. For instance, in periodic signals (such as Cosine or a square wave) there are infinite correct answers since the real phase shift + integer times of the period will give the same result. On the other hand, even in non-periodic signals, such as one impulse, noise may effect the phase detection, in some cases more than others. This field is broad and the answer strongly depend on what you want to achieve and the data quality.

Having said that, here is my output from your data (two components) using ACF as a first step for the inspection, I don't see a problem with it:

EDIT:
Added some corrections and changed signals to the real targets.

ANOTHER EDIT:

Regarding phase shift estimation, there are many ways to do so. Please research the traditional literature for answers and explore techniques oriented for your type of data. My suggestions:

1. First peak of ACF can be a good answer.
2. Projection on sine waves (partial Fourier series) and adjusting the phase at the largest coefficients.
3. Phase detection based on fitting the same models and inspection of parameters (can be done automatic, depends on the model).
4. Using an envelope detector (for this data) and detecting the phases between the two upper lines and the two bottom lines. You'll get two estimations, you can take the average.
5. Use more preprocessing - trend and seasonality decomposition, and take the seasonality component for inspecting the phase.

There are many ways, these just pop up right away. I'm sure that by playing with it a little you can find that suits your data in a reasonable computation time so it will be applicable for your product.

Good luck!

Here is the code:

# just some imports
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

plt.style.use('ggplot')

# load data
df = pd.read_csv('c:\\Temp\\curve_fitting_ahmed.csv')

# normalization before ACF
def normalize(data):
    return (data - np.mean(data, axis=0).reshape((1, -11))) / (np.std(data, axis=0).reshape((1, -1)))


# select subset of columns, seems relevant as a group
SCOLS = ['TW_A1','W_A1']

# just to see the data
f = plt.figure()
ax = f.add_subplot(111)
df[SCOLS[:2]].iloc[::10].plot(ax=ax)
ax.set_title('Raw data')

# normalization
normalized = normalize(df[SCOLS].values)
f = plt.figure()
ax = f.add_subplot(111)
ax.plot(np.arange(normalized.shape[0]),normalized[:,0], label='TW_A1')
ax.plot(np.arange(normalized.shape[0]),normalized[:,1], label='W_A1')
ax.set_title('Normalized')

# ACF between two components
x1x2 = np.correlate(normalized[:, 0], normalized[:, 1], 'full')

# see the results
f = plt.figure()
ax = f.add_subplot(111)
ax.plot(x1x2)
ax.set_title('ACF')

# mainloop
plt.show()