CODEWITHSUNDEEP
pythonpandasdataframe
techvslife
techvslife

Reputation: 2373

Involuntary conversion of int64 to float64 in pandas

Never mind the below--I see the cause of the problem. The shift of course produces a N/A.

I want to prevent a type conversion that occurs when concatenating a dataframe to itself horizontally. I have a dataframe where all columns are int64 (and the index is a datetime64[ns]):

df.dtypes
Out[118]: 
op        int64

I concatenate to have the next row's columns (suffixed with "_next") appear on the same line as the current row: 

df = pd.concat([df, df.shift(-1).add_suffix('_next')], axis=1)

But then the types on the concatenated columns change to float64:

df.dtypes
Out[122]: 
op               int64
op_next        float64

Is there a way to prevent that type conversion? Thanks.

Upvotes: 1

Views: 1198

Answers (2)

jpp
jpp

Reputation: 164623

This occurs because df.shift(-1) has one element which is NaN, which is a float. Such a series will automatically be upcasted to float. Here is a minimal example:

df = pd.DataFrame({'op': [1, 2, 3]})
df = pd.concat([df, df.shift(-1).add_suffix('_next')], axis=1)

print(df)

   op  op_next
0   1      2.0
1   2      3.0
2   3      NaN

There is nothing you can do except use fillna to fill with an integer and recast. You can do this either before or after pd.concat:

Before

df = pd.concat([df, df.shift(-1).fillna(0).astype(int).add_suffix('_next')], axis=1)

After

df = pd.concat([df, df.shift(-1).add_suffix('_next')], axis=1)
df = df.fillna(0).astype(int)

Upvotes: 3

techvslife
techvslife

Reputation: 2373

Shift produces a n/a value, forcing the float64.

Upvotes: 1

Related Questions