CODEWITHSUNDEEP
pythoncsvpandas
Andy
Andy

Reputation: 50640

How do I prevent pandas from recasting my integer values to floats?

I am merging two CSV files with input below.

file1.csv

Id,attr1,attr2,attr3
1,True,7,"Purple"
2,False,19.8,"Cucumber"
3,False,-0.5,"A string with a comma, because it has one"
4,True,2,"Nope"
5,True,4.0,"Tuesday"
6,False,1,"Failure"

file2.csv

Id,attr4,attr5,attr6
2,"python",500000.12,False
5,"program",3,True
3,"Another string",-5,False

When I run this code

import pandas as pd
df1 = pd.read_csv("file1.csv")
df2 = pd.read_csv("file2.csv")
merged = df1.merge(df2, on="Id", how="outer").fillna("")
merged.to_csv("merged.csv", index=False)

I get output like this

Id,attr1,attr2,attr3,attr4,attr5,attr6
1,True,7.0,Purple,,,
2,False,19.8,Cucumber,python,500000.12,False
3,False,-0.5,"A string with a comma, because it has one",Another string,-5.0,False
4,True,2.0,Nope,,,
5,True,4.0,Tuesday,program,3.0,True
6,False,1.0,Failure,,,

Notice that attr2 from several of my records has been converted from an int to a float. 

1,True,7.0,Purple,,,

vs the expected

1,True,7,Purple,,,

For this example data set it is a minor annoyance. However, when I am running it against my large set of data, this behavior is occuring on my Id column as well. This would break processes further down my workflow chain.

How do I prevent pandas from doing this conversion either for the entire file or, ideally, for specific columns?

Upvotes: 4

Views: 7811

Answers (1)

DSM
DSM

Reputation: 353569

You can pass a value (whether a type if you want to affect the entire DataFrame or a dictionary if you want to affect individual columns) to the dtype argument:

>>> df = pd.read_csv("file1.csv", dtype={"id": int, "attr2": str})
>>> df
   id  attr1 attr2                                      attr3
0   1   True     7                                     Purple
1   2  False  19.8                                   Cucumber
2   3  False  -0.5  A string with a comma, because it has one
3   4   True     2                                       Nope
4   5   True   4.0                                    Tuesday
5   6  False     1                                    Failure

[6 rows x 4 columns]
>>> df.dtypes
id        int32
attr1      bool
attr2    object
attr3    object
dtype: object

Upvotes: 7

Related Questions