CODEWITHSUNDEEP
pythonpandaserror-handlingbin
jeangelj
jeangelj

Reputation: 4498

Python ValueError: Bin edges must be unique

Working in python with pandas, I am trying to assign control and treatment groups to different groups of customers.

I have a large dataset. Instead of giving an example of the data, let me show you the pivot, since this summarizes the most important data. 

pd.pivot_table(df,index=['Test Group'],values=["Customer_ID"],aggfunc=lambda x: len(x.unique()))

I get those counts Test Group Customer_ID 

Innovators 4634
Early Adopters 2622
Early Majority 8653
Late Majority 7645
Laggards 7645
Lost 4354
Prospective 653

I run the following code:

percentages = {'Innovators':[0.0,1.0],\
     'Early Adopters':[0.2,0.8], \
     'Early Majority':[0.1,0.9],\
     'Late Majority':[0.0,1.0],\
     'Laggards':[0.2,0.8],\
     'Lost':[0.1,0.9],\
     'Prospective':[0.1,0.9]}

def assigner(gp):
     ...:     group = gp['Test Group'].iloc[0]
     ...:     cut = pd.qcut(
                  np.arange(gp.shape[0]), 
                  q=np.cumsum([0] + percentages[group]), 
                  labels=range(len(percentages[group]))
              ).get_values()
     ...:     return pd.Series(cut[np.random.permutation(gp.shape[0])], index=gp.index, name='flag')

df['flag'] = df.groupby('Test Group', group_keys=False).apply(assigner)

ValueError: Bin edges must be unique: array([   0,    0, 2621], dtype=int64).
You can drop duplicate edges by setting the 'duplicates' kwarg

... and keep on getting this error

I found this answer, which could be helpful How to qcut with non unique bin edges? ; but rank dowsn't work for np

def assigner(gp):
     ...:     group = gp['Campaign Test Description'].iloc[0]
     ...:     cut = pd.qcut(
                  np.arange(gp.shape[0]).rank(method='first'), 
                  q=np.cumsum([0] + percentages[group]), 
                  labels=range(len(percentages[group]))
              ).get_values()
     ...:     return pd.Series(cut[np.random.permutation(gp.shape[0])], index=gp.index, name='flag')

AttributeError: 'numpy.ndarray' object has no attribute 'rank'

I tried dropping duplicates

def assigner(gp):
     ...:     group = gp['Campaign Test Description'].iloc[0]
     ...:     cut = pd.qcut(
                  np.arange(gp.shape[0]), 
                  q=np.cumsum([0] + percentages[group]), 
                  labels=range(len(percentages[group])),duplicates='drop'
              ).get_values()
     ...:     return pd.Series(cut[np.random.permutation(gp.shape[0])], index=gp.index, name='flag')

ValueError: Bin labels must be one fewer than the number of bin edges

Still getting an error

Upvotes: 0

Views: 2943

Answers (1)

Stev
Stev

Reputation: 1150

You are doing a train/test split, which is commonly used in machine learning. Here is a way to do it (double check that I have your percentages the right way around):

df_pct = pd.DataFrame({ 'ID': ['Innovators','Early Adopters' ,'Early Majority','Late Majority','Laggards','Lost','Prospective'], 'test_cutoff':[1,0.8,0.9,0.1,0.8,0.9,0.9]}) df=df.merge(df_pct) df['is_test'] = np.random.uniform(0, 1, len(df)) >= df['test_cutoff']

Also, your 'Late Majority' percentages don't add up to 100.

Upvotes: 1

Related Questions