Phase shift between X and Y components of the same wave

Question

Not sure if this is the right place/format to ask this.

I have two signals, I1 and I2, which were acquired from a white light source in a spectrally resolved interferometer with a waveplate and a polarizer in one arm, such that they are the X and Y components of the original wave after introducing the shift, and theoretically they should have a phase difference of pi/2. I'm trying to extract said phase difference between the two signals using FFT. I have been trying for a few days and I'm sure this is not that hard and there's something I'm just not understanding about either Fourier transform or the nature of my signals, but I could use some help to realize what that is. Once again this if this is not the place to ask this I will simply delete the post and keep trying.

I have tried a lot of approaches along the lines of this, which is the code I currently have:

k = np.fft.rfftfreq(I1.size, d=num_onda_equi[1]-num_onda_equi[0])
mask = k > 0
F1 = np.fft.rfft(I1)[mask]
F2 = np.fft.rfft(I2)[mask]
fase1 = np.angle(F1)
fase2 = np.angle(F2)
conv = np.conj(F1) * F2


plt.figure(figsize=(12, 16))

# Señales del interferómetro
plt.subplot(3, 1, 1)
plt.plot(I1, label='I1')
plt.plot(I2, label='I2')
plt.title('Intensidades')
plt.legend()

# Fases individuales
plt.subplot(3, 1, 2)
plt.plot(fase1, label='Fase X')
plt.plot(fase2, label='Fase Y')
plt.legend()
plt.title('Fases')
plt.legend()

# Diferencia de fases
plt.subplot(3, 1, 3)
plt.plot(np.angle(conv), 'g-')
plt.title('Diferencia de fases')
plt.show()   

This produces the following result:

1. Original signals, the phase difference between them isn't constant for some reason but should be around pi/2 for most values. 2. Phase from both signals, just to see if they make sense. 3. Phase difference.

I expected to get a somewhat constant plot for the phase difference around pi/2 since that's the only phase difference the X and Y components are supposed to have. I have also tried using Hilbert transforms adapting the code from this link without success, and checked a lot of similar questions but no answer has led me to the expected result.