CODEWITHSUNDEEP
machine-learningscikit-learn
J Ng
J Ng

Reputation: 799

Recursive Feature Elimination (RFE) SKLearn

I created a table to test my understanding

    F1  F2  Outcome
0   2   5        1
1   4   8        2
2   6   0        3
3   9   8        4
4  10   6        5

From F1 and F2 I tried to predict Outcome

As you can see F1 have a strong correlation to Outcome,F2 is random noise

I tested

pca = PCA(n_components=2)
fit = pca.fit(X)
print("Explained Variance")
print(fit.explained_variance_ratio_)
Explained Variance
[ 0.57554896  0.42445104]

Which is what I expected and shows that F1 is more important

However when I do RFE (Recursive Feature Elimination)

model = LogisticRegression()
rfe = RFE(model, 1)
fit = rfe.fit(X, Y)
print(fit.n_features_)
print(fit.support_)
print(fit.ranking_)
1
[False  True]
[2 1]

It asked me to keep F2 instead? It should ask me to keep F1 since F1 is a strong predictor while F2 is random noise... why F2?

Thanks

Upvotes: 0

Views: 2396

Answers (2)

The_Data_Guy
The_Data_Guy

Reputation: 193

It is advisable to do a Recursive Feature Elimination Cross Validation (RFECV) before running the Recursive Feature Elimination (RFE)

Here is an example: Having columns :

df.columns = ['age', 'id', 'sex', 'height', 'gender', 'marital status', 'income', 'race']

Use RFECV to identify the optimal number of features needed.

from sklearn.ensemble import RandomForestClassifier
rfe = RandomForestClassifier(random_state = 32) # Instantiate the algo
rfecv  = RFECV(estimator= rfe, step=1, cv=StratifiedKFold(2), scoring="accuracy") # Instantiate the RFECV and its parameters
fit = rfecv.fit(features(or X), target(or y))
print("Optimal number of features : %d" % rfecv.n_features_)
>>>> Optimal number of output is 4

Now that the optimal number of features has been known, we can use Recursive Feature Elimination to identify the Optimal features

from sklearn.feature_selection import RFE
min_features_to_select = 1
rfc = RandomForestClassifier()
rfe = RFE(estimator=rfc, n_features_to_select= 4, step=1)
fittings1 = rfe.fit(features, target)
for i in range(features.shape[1]):
    print('Column: %d, Selected %s, Rank: %.3f' % (i, rfe.support_[i], rfe.ranking_[i]))

output will be something like:
>>> Column: 0, Selected True, Rank: 1.000
>>> Column: 1, Selected False, Rank: 4.000
>>> Column: 2, Selected False, Rank: 7.000
>>> Column: 3, Selected False, Rank: 10.000
>>> Column: 4, Selected True, Rank: 1.000
>>> Column: 5, Selected False, Rank: 3.000

Now display the features to remove based on recursive feature elimination done above

columns_to_remove = features.columns.values[np.logical_not(rfe.support_)]
columns_to_remove

output will be something like:
>>> array(['age', 'id', 'race'], dtype=object)

Now create your new dataset by dropping the un-needed features and selecting the needed one

new_df = df.drop(['age', 'id', 'race'], axis = 1)

Then you can cross validation to know how well this newly selected features (new_df) predicts the target column.

# Check how well the features predict the target variable using cross_validation
cv = ShuffleSplit(n_splits=5, test_size=0.3, random_state=0)
scores = cross_val_score(RandomForestClassifier(), new_df, target, cv= cv)

print("%0.2f accuracy with a standard deviation of %0.2f" % (scores.mean(), scores.std()))
>>> 0.84 accuracy with a standard deviation of 0.01

Don't forget you can also read up on the best Cross-Validation (CV) parameters to use in this documentation

Recursive Feature Elimination (RFE) documentation to learn more and understand better

Recursive Feature Elimination Cross validation (RFECV) documentation

Upvotes: 5

Vivek Kumar
Vivek Kumar

Reputation: 36599

You are using LogisticRegression model. This is a classifier, not a regressor. So your outcome here is treated as labels (not numbers). For good training and prediction, a classifier needs multiple samples of each class. But in your data, only single row is present for each class. Hence the results are garbage and not to be taken seriously.

Try replacing that with any regression model and you will see the outcome which you thought would be.

model = LinearRegression()
rfe = RFE(model, 1)
fit = rfe.fit(X, y)

print(fit.n_features_)
print(fit.support_)
print(fit.ranking_)

# Output
1
[ True False]
[1 2]

Upvotes: 2

Related Questions