CODEWITHSUNDEEP
pythonpandasscipyinterpolation
Darkstarone
Darkstarone

Reputation: 4730

Pandas interpolation not interpolating with certain methods

I have a dataframe like so:

[5232 rows x 2 columns]
                                       0       2
0                                               
2018-02-01 00:00:00  2018-02-01 00:00:00  435.24
2018-02-01 00:30:00  2018-02-01 00:30:00     NaN
2018-02-01 01:00:00  2018-02-01 01:00:00  301.32
2018-02-01 01:30:00  2018-02-01 01:30:00  256.68
2018-02-01 02:00:00  2018-02-01 02:00:00  245.52

And I'm trying to interpolate it. I've found that the basic panda's methods work fine (e.g. time or linear), but if I try and use a scipy method like krogh or barycentric I find that the interpolation doesn't appear to interpolate the points:

                                          0       2
0                                               
2018-02-01 00:00:00  2018-02-01 00:00:00  435.24
2018-02-01 00:30:00  2018-02-01 00:30:00     NaN
2018-02-01 01:00:00  2018-02-01 01:00:00  301.32
2018-02-01 01:30:00  2018-02-01 01:30:00  256.68
2018-02-01 02:00:00  2018-02-01 02:00:00  245.52

My interpolation method is as follows:

def interpolate(df : DataFrame, interpolate_type : str = 'pandas'):
    """ Helper method for inserting different interpolation methods into the main function. """
    if interpolate_type == 'pandas':
        return df.interpolate(limit_direction='both', method='time') 
    if interpolate_type == 'krogh':
        return df.interpolate(limit_direction='both', method='krogh')

Is there something else you need to do to get scipy interpolation methods to work?

Edit: Here is the file I'm working on: link

Also, here is my toy script that fails with the above CSV:

df_2[2] = pd.to_numeric(df_2[2],errors='force')
df_2 = df_2.set_index(pd.DatetimeIndex(df_2[0])) # Increases interpolation accuracy.
df_2.index = pd.to_datetime(df_2.index)
df_2.iloc[1, 2] = np.NaN
df_2.sort_index(inplace=True)

print(df_2.interpolate(limit_direction='both', method='krogh'))

If I change the interpolate function to anyscipy` version it fails.

Also here is my toy case that's failing on real data:

Upvotes: 3

Views: 1758

Answers (1)

Grr
Grr

Reputation: 16079

Given your sample data (formatted with DatetimeIndex) all available methods appear to work with pandas 0.22.0:

                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00     NaN
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52

df.info()
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 5 entries, 2018-02-01 00:00:00 to 2018-02-01 02:00:00
Data columns (total 2 columns):
0    5 non-null datetime64[ns]
1    4 non-null float64
dtypes: datetime64[ns](1), float64(1)
memory usage: 120.0 bytes

methods = ['linear', 'time', 'index', 'values', 'nearest', 'zero',
           'slinear', 'quadratic', 'cubic', 'barycentric', 'krogh', 
           'piecewise_polynomial', 'from_derivatives', 'pchip', 'akima']

for method in methods:
    print(method)
    print(df.interpolate(limit_direction='both', method=method))

linear
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  368.28
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
time
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  368.28
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
index
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  368.28
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
values
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  368.28
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
nearest
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  435.24
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
zero
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  435.24
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
slinear
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  368.28
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
quadratic
                                      0           1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.240000
2018-02-01 00:30:00 2018-02-01 00:30:00  361.818947
2018-02-01 01:00:00 2018-02-01 01:00:00  301.320000
2018-02-01 01:30:00 2018-02-01 01:30:00  256.680000
2018-02-01 02:00:00 2018-02-01 02:00:00  245.520000
cubic
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  365.49
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
barycentric
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  245.52
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
krogh
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  365.49
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
piecewise_polynomial
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  368.28
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
from_derivatives
                                      0       1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24
2018-02-01 00:30:00 2018-02-01 00:30:00  368.28
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52
pchip
                                      0          1
2018-02-01 00:00:00 2018-02-01 00:00:00  435.24000
2018-02-01 00:30:00 2018-02-01 00:30:00  360.92087
2018-02-01 01:00:00 2018-02-01 01:00:00  301.32000
2018-02-01 01:30:00 2018-02-01 01:30:00  256.68000
2018-02-01 02:00:00 2018-02-01 02:00:00  245.52000
akima
                                      0             1
2018-02-01 00:00:00 2018-02-01 00:00:00  4.352400e+02
2018-02-01 00:30:00 2018-02-01 00:30:00 -5.045003e+07
2018-02-01 01:00:00 2018-02-01 01:00:00  3.013200e+02
2018-02-01 01:30:00 2018-02-01 01:30:00  2.566800e+02
2018-02-01 02:00:00 2018-02-01 02:00:00  2.455200e+02

Upvotes: 3

Related Questions