CODEWITHSUNDEEP
pythonpandasdataframeseries
The L
The L

Reputation: 126

Avoid SettingWithCopyWarning in python using iloc

Usually, to avoid SettingWithCopyWarning, I replace values using .loc or .iloc, but this does not work when I want to forward fill my column (from the first to the last non-nan value). Do you know why it does that and how to bypass it ?

My test dataframe :

df3 = pd.DataFrame({'Timestamp':[11.1,11.2,11.3,11.4,11.5,11.6,11.7,11.8,11.9,12.0,12.10,12.2,12.3,12.4,12.5,12.6,12.7,12.8,12.9],
                   'test':[np.nan,np.nan,np.nan,2,22,8,np.nan,4,5,4,5,np.nan,-3,-54,-23,np.nan,89,np.nan,np.nan]})

and the code that raises me a warning :

df3['test'].iloc[df3['test'].first_valid_index():df3['test'].last_valid_index()+1] = df3['test'].iloc[df3['test'].first_valid_index():df3['test'].last_valid_index()+1].fillna(method="ffill")

I would like something like that in the end :

enter image description here

Upvotes: 0

Views: 120

Answers (1)

user8560167
user8560167

Reputation:

Use first_valid_index and last_valid_index to determine range that you want to ffill and then select range of your dataframe

df = pd.DataFrame({'Timestamp':[11.1,11.2,11.3,11.4,11.5,11.6,11.7,11.8,11.9,12.0,12.10,12.2,12.3,12.4,12.5,12.6,12.7,12.8,12.9],
                   'test':[np.nan,np.nan,np.nan,2,22,8,np.nan,4,5,4,5,np.nan,-3,-54,-23,np.nan,89,np.nan,np.nan]})

first=df['test'].first_valid_index()
last=df['test'].last_valid_index()+1

df['test']=df['test'][first:last].ffill()
print(df)
    Timestamp  test
0        11.1   NaN
1        11.2   NaN
2        11.3   NaN
3        11.4   2.0
4        11.5  22.0
5        11.6   8.0
6        11.7   8.0
7        11.8   4.0
8        11.9   5.0
9        12.0   4.0
10       12.1   5.0
11       12.2   5.0
12       12.3  -3.0
13       12.4 -54.0
14       12.5 -23.0
15       12.6 -23.0
16       12.7  89.0
17       12.8   NaN
18       12.9   NaN

Upvotes: 1

Related Questions