CODEWITHSUNDEEP
pythonpandasnumpy
user3226167
user3226167

Reputation: 3449

pandas shift converts my column from integer to float

shift converts my column from integer to float. It turns out that np.nan is float only. Is there a way to keep the shifted column as integers?

df = pd.DataFrame({"a":range(5)})
df['b'] = df['a'].shift(1)

df['a']
# 0    0
# 1    1
# 2    2
# 3    3
# 4    4
# Name: a, dtype: int64

df['b']

# 0   NaN
# 1     0
# 2     1
# 3     2
# 4     3
# Name: b, dtype: float64

Upvotes: 23

Views: 10306

Answers (6)

Mahesh
Mahesh

Reputation: 141

Another solution is to use the replace() function and typecast:

df['b'] = df['a'].shift(1).replace(np.NaN,0).astype(int)

Upvotes: 0

jezrael
jezrael

Reputation: 863301

A solution for pandas under 0.24:

The problem is you get a NaN value that is float, so int is converted to float - see na type promotions.

One possible solution is convert NaN values to some value like 0 and then it is possible convert to int:

df = pd.DataFrame({"a":range(5)})
df['b'] = df['a'].shift(1).fillna(0).astype(int)
print (df)
   a  b
0  0  0
1  1  0
2  2  1
3  3  2
4  4  3

A solution for pandas 0.24+ - check Series.shift:

fill_value object, optional The scalar value to use for newly introduced missing values. the default depends on the dtype of self. For numeric data, np.nan is used. For datetime, timedelta, or period data, etc. NaT is used. For extension dtypes, self.dtype.na_value is used.

Changed in version 0.24.0.

df['b'] = df['a'].shift(fill_value=0)

Upvotes: 18

piRSquared
piRSquared

Reputation: 294508

You can construct a NumPy array by prepending a 0 to all but the last element of column a:

df.assign(b=np.append(0, df.a.values[:-1]))

   a  b
0  0  0
1  1  0
2  2  1
3  3  2
4  4  3

Upvotes: 4

David
David

Reputation: 86

Another solution starting from pandas version 0.24.0: simply provide a value for the parameter fill_value:

df['b'] = df['a'].shift(1, fill_value=0)

Upvotes: 7

Mithril
Mithril

Reputation: 13778

I don't like other answers which may change original dtypes. What if you have float or str in the data?

Since we don't need the first nan row, why not skip it?

I would keep all dtypes and cast back:

dt = df.dtypes
df = df.shift(1).iloc[1:].astype(dt)

Upvotes: 0

totalhack
totalhack

Reputation: 2608

As of pandas 1.0.0 I believe you have another option, which is to first use convert_dtypes. This converts the dataframe columns to dtypes that support pd.NA, avoiding the issues with NaN. 

df = pd.DataFrame({"a":range(5)})
df = df.convert_dtypes()
df['b'] = df['a'].shift(1)

print(df['a'])
# 0    0
# 1    1
# 2    2
# 3    3
# 4    4
# Name: a, dtype: Int64

print(df['b'])
# 0    <NA>
# 1       0
# 2       1
# 3       2
# 4       3
# Name: b, dtype: Int64

Upvotes: 4

Related Questions