CODEWITHSUNDEEP
pythonpandasimbalanced-dataoversamplingsmote
psagrera
psagrera

Reputation: 141

How to keep/extend index when oversample

I've got a dataframe like that , and I want to oversample the column "role" (in a real case the number of rows/columns in much bigger than this minimal example)

                 role  value
pop_13vdpn1_site_1  1   1
pop_13vdpn1_site_1  1   1
pop_13vdpn1_site_1  1   2
pop_13vdpn1_site_1  1   1
pop_13vdpn1_site_1  1   1
pop_13vdpn1_site_1  1   2
pop_13vdpn1_site_1  1   1
pop_13vdpn1_site_1  2   1
pop_13vdpn1_site_1  2   1
pop_13vdpn1_site_1  2   1
pop_13vdpn1_site_2  2   1
pop_13vdpn1_site_2  2   2
pop_13vdpn1_site_2  2   1
pop_13vdpn1_site_2  2   1
pop_13vdpn1_site_2  2   1
pop_13vdpn1_site_2  2   1
pop_13vdpn1_site_2  2   1
pop_13vdpn1_site_2  2   1
pop_13vdpn1_site_2  2   1
pop_13vdpn1_site_3  2   1
[...........]

Index: 20 entries, pop_13vdpn1_site_1 to pop_13vdpn1_site_1
Data columns (total 2 columns):
role     20 non-null int64
value    20 non-null int64

That's what I'm doing :

X,y = smote.fit_sample(df,df[['role']])
X
       role value
0   1   1
1   1   1
2   1   2
3   1   1
4   1   1
5   1   2
6   1   1
7   2   1
8   2   1
[.........]

and it works, but the problem is that I need to keep the index (pop_13vdpn1_site_1, etc..) is that possible ?

Upvotes: 0

Views: 1312

Answers (3)

Ruthger Righart
Ruthger Righart

Reputation: 4921

The following should do it.

import io
import pandas as pd
import numpy as np
from imblearn.over_sampling import SMOTE

Example data.

df = pd.read_csv(io.StringIO("""
role  value
    pop_13vdpn1_site_1 1 1
    pop_13vdpn1_site_1 1 1
    pop_13vdpn1_site_1 1 2
    pop_13vdpn1_site_1 1 1
    pop_13vdpn1_site_1 1 1
    pop_13vdpn1_site_1 1 2
    pop_13vdpn1_site_1 1 1
    pop_13vdpn1_site_1 2 1
    pop_13vdpn1_site_1 2 1
    pop_13vdpn1_site_1 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 2
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_3 2 1
    pop_13vdpn1_site_1 1 1
    pop_13vdpn1_site_1 1 1
    pop_13vdpn1_site_1 1 2
    pop_13vdpn1_site_1 1 1
    pop_13vdpn1_site_1 1 1
    pop_13vdpn1_site_1 1 2
    pop_13vdpn1_site_1 1 1
    pop_13vdpn1_site_1 2 1
    pop_13vdpn1_site_1 2 1
    pop_13vdpn1_site_1 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 2
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_2 2 1
    pop_13vdpn1_site_3 2 1
"""), sep="\s+", engine="python")

df = df.reset_index()

Shape should be (40, 3):

df.shape

Smote accept arrays, so we need to define the x and y values.

X_train = np.array(df['role']).reshape(40,1)
y_train = np.array(df['value']).reshape(40,)

Smote in action:

from imblearn.over_sampling import SMOTE
sm = SMOTE(random_state=42)
X,y = sm.fit_resample(X_train,y_train)

Put the given X and y in a DataFrame:

ndf = pd.DataFrame({'role':X.reshape(68,), 'value':y})

Remake the original names.

ndf['name'] = ndf['role'].apply(lambda x: 'pop_13vdpn1_site_'+str(x))

To see if the data are more balanced.

from collections import Counter
Counter(df['role'])
Counter(ndf['role'])

Upvotes: 0

psagrera
psagrera

Reputation: 141

Finally I've found a workaround (Maybe not optimal)

from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df_tmp = df.reset_index()
df_tmp['index'] = le.fit_transform(df_tmp['index'])
aa,bb = smote.fit_sample(df_tmp,df_tmp[['role']])
aa['index'] = le.inverse_transform(aa['index'])
aa.set_index('index')

Upvotes: 1

Giorgos Myrianthous
Giorgos Myrianthous

Reputation: 39950

First of all you need process df and split your features and target labels as X_train and y_train.

Now you can do your oversampling:

X_train_over, y_train_over = smote.fit_sample(X_train, y_train)

and finally create a dataframe from the above output. For example,

X = pd.DataFrame(X_train_over, columns=X_train.columns)
y = pd.DataFrame(y_train_over, columns=y_train.columns)

Upvotes: 0

Related Questions