CODEWITHSUNDEEP
pythonpandas
jason m
jason m

Reputation: 6835

pandas df.corr() returns NaN despite data fed having populated data

My data looks like the following:

print data

                   A                B 
2014-04-04        163.24         191.77
2014-06-11        165.43         182.25
2014-12-22        194.44         161.44
2014-03-25        163.25         195.04
2014-11-03        190.83         164.36
2014-06-16        165.85         182.35
2014-11-24        190.07         162.15
2014-05-21        159.35         186.39
2015-01-21        177.15         152.09
2014-08-28        177.65         192.00
2014-02-19        163.26         182.95
2014-05-12        159.55         192.57
2014-07-09        164.67         188.42
2015-02-26        192.20         160.87
2014-10-14        178.70         183.80
2014-07-16        170.47         192.36
2014-01-21        173.20         188.43
2014-06-03        161.80         184.37
2014-03-17        166.84         185.81
2014-08-11        172.46         187.47
2015-01-07        187.28         155.05
2014-05-29        160.74         183.76
2015-02-11        187.65         158.20
2014-02-27        165.38         185.27
2015-01-05        188.34         159.51
2014-07-01        166.81         186.35
2015-02-18        188.67         162.19
2014-04-03        166.09         192.69
2014-10-06        187.48         189.04
2014-12-29        196.19         160.51

When I try to call data.corr() I receive back all NaN. I tested this in excel, and the correlation is indeed able to be calculated.

Any idea why this might be the case?

I am happy to post more but i am unsure what would help.

print data.info()

<class 'pandas.core.frame.DataFrame'>
Index: 298 entries, 2014-04-04 to 2014-01-29
Data columns (total 2 columns):
A     298 non-null float64
B    298 non-null float64
dtypes: float64(2)None



print data.describe()
              A                B
    count    298.000000     298.000000
    mean     175.152886     178.596242
    std       11.490668      13.709316
    min      152.720000     151.410000
    25%      165.502500     162.655000
    50%      175.480000     184.330000
    75%      185.085000     189.692500
    max      196.890000     197.770000


print pd.version
0.14.1

Upvotes: 8

Views: 21775

Answers (2)

Jigidi Sarnath
Jigidi Sarnath

Reputation: 66

I encountered a similar problem when "s1.corr(s2)" was returning NaN. Both s1 and s2 are pandas series. The issue turned out to be different data-type for the "index". s1 had datetime as index whereas s2 had "date" as index. This was not visible when I print these series in Jupyter notebook. Both of them were showing the index as "date".... I found the issue when I concatenated these series to make 2 columns of a dataframe and then the result revealed the difference... Bottomline: If you encounter this with pandas series, then look at the index. Dont go with just print log of the index. Try concatenating them as columns of a dataframe and you can clearly see the issue

Upvotes: 0

EngineeredE
EngineeredE

Reputation: 741

Jason, Using the sample data it works fine for me.

data.corr()
    A   B
A   1.000000    -0.779864
B   -0.779864   1.000000

I noticed in your data.info() there is a 'None' tacked on to dtypes. When I load your sample data I get

data.info()
<class 'pandas.core.frame.DataFrame'>
Index: 30 entries, 2014-04-04 to 2014-12-29
Data columns (total 2 columns):
A     30 non-null float64
B     30 non-null float64
dtypes: float64(2)

I imagine what's causing that 'None' is causing your problem.

Hope this helps [wish I could leave as a comment but don't have the reputation yet!]

Upvotes: 9

Related Questions