CODEWITHSUNDEEP
pythonnumpyscipypandasstatsmodels
user248237
user248237

Reputation:

Getting CDF of variable-sized numpy arrays in Python using same bins?

I'd like to make a set of comparable empirical CDFs for a few numpy arrays (each of different length) and store these in a pandas dataframe:

a = scipy.randn(100)
b = scipy.randn(500)
# ECDF from statmodels
cdf_a = ECDF(a)
cdf_b = ECDF(b)

The problem is that cdf_a.x, cdf_a.y will be of different lengths of cdf_b.x, cdf_b.y and I would like these to be the same length, i.e. use same number of bins to compute the CDF so that these can be plotted on same scale from a pandas DataFrame. This is not possible:

df = pandas.DataFrame({"cdf_a": cdf_a.y, "cdf_b": cdf_b.y})

Since the cdfs are not of the same length. How can I bin a and b using similar bins when computing their CDFs, so that I get comparable same-length vectors back?

Is this the best solution?

bins = np.linspace(0, 1, 10)
v1 = cdf_a(bins)
v2 = cdf_b(bins)

Upvotes: 2

Views: 606

Answers (2)

Josef
Josef

Reputation: 22897

The way we use it in some goodness of fit tests is to stack the arrays, so they are defined on all points, points from both arrays.

Then use np.searchsorted to get the ranking, number of points in dataset 1 below x and number of points in dataset 2 below x.

If I remember correctly, look at scipy.stats.ks_2samp

data1 = np.sort(data1)
data2 = np.sort(data2)
data_all = np.concatenate([data1,data2])
cdf1 = np.searchsorted(data1,data_all,side='right')/(1.0*n1)
cdf2 = (np.searchsorted(data2,data_all,side='right'))/(1.0*n2)

Upvotes: 1

user248237
user248237

Reputation:

It appears that this is a good solution:

bins = np.linspace(0, 1, 10)
v1 = cdf_a(bins)
v2 = cdf_b(bins)

Then len(v1) == len(v2) and these can be plotted as CDFs of a, b on the same scale.

Upvotes: 0

Related Questions