CODEWITHSUNDEEP
pythonmatplotlibscipy
user8834780
user8834780

Reputation: 1670

PDF plotting concern

I tried the following manual approach:

dict = {'id': ['a','b','c','d'], 'testers_time': [10, 30, 15, None], 'stage_1_to_2_time': [30, None, 30, None], 'activated_time' : [40, None, 45, None],'stage_2_to_3_time' : [30, None, None, None],'engaged_time' : [70, None, None, None]} 
df = pd.DataFrame(dict, columns=['id', 'testers_time', 'stage_1_to_2_time', 'activated_time', 'stage_2_to_3_time', 'engaged_time'])

df= df.dropna(subset=['testers_time']).sort_values('testers_time')

prob = df['testers_time'].value_counts(normalize=True)
print(prob)
#0.333333,  0.333333,  0.333333
plt.plot(df['testers_time'], prob, marker='.', linestyle='-') 

plt.show()

And I tried the following approach I found on stackoverflow:

dict = {'id': ['a','b','c','d'], 'testers_time': [10, 30, 15, None], 'stage_1_to_2_time': [30, None, 30, None], 'activated_time' : [40, None, 45, None],'stage_2_to_3_time' : [30, None, None, None],'engaged_time' : [70, None, None, None]} 
df = pd.DataFrame(dict, columns=['id', 'testers_time', 'stage_1_to_2_time', 'activated_time', 'stage_2_to_3_time', 'engaged_time'])

df= df.dropna(subset=['testers_time']).sort_values('testers_time')

fit = stats.norm.pdf(df['testers_time'], np.mean(df['testers_time']), np.std(df['testers_time']))  
print(fit)
#0.02902547,  0.04346777,  0.01829513]
plt.plot(df['testers_time'], fit, marker='.', linestyle='-')
plt.hist(df['testers_time'], normed='true')      

plt.show()

As you can see I get completely different values- the probabilities are correct for #1, but for #2 they aren't (nor do they add up to 100%), and the y axis (%) of the histogram is based on 6 bins, not 3.

Can you explain how I can get the right probability for #2?

Upvotes: 0

Views: 53

Answers (1)

ImportanceOfBeingErnest
ImportanceOfBeingErnest

Reputation: 339260

The first approach gives you a probability mass function. The second gives you a probability density - hence the name probability density function (pdf). Hence both are correct, they just show something different.

If you evaluate the pdf over a larger range (e.g. 10 times the standard deviation), it will look much like an expected gaussian curve.

import pandas as pd
import scipy.stats as stats
import numpy as np
import matplotlib.pyplot as plt

dict = {'id': ['a','b','c','d'], 'testers_time': [10, 30, 15, None], 'stage_1_to_2_time': [30, None, 30, None], 'activated_time' : [40, None, 45, None],'stage_2_to_3_time' : [30, None, None, None],'engaged_time' : [70, None, None, None]} 
df = pd.DataFrame(dict, columns=['id', 'testers_time', 'stage_1_to_2_time', 'activated_time', 'stage_2_to_3_time', 'engaged_time'])

df= df.dropna(subset=['testers_time']).sort_values('testers_time')

mean = np.mean(df['testers_time'])
std = np.std(df['testers_time'])
x = np.linspace(mean - 5*std, mean + 5*std)

fit = stats.norm.pdf(x, mean, std)  
print(fit)

plt.plot(x, fit, marker='.', linestyle='-')
plt.hist(df['testers_time'], normed='true')      

plt.show()

enter image description here

Upvotes: 1

Related Questions