CODEWITHSUNDEEP
pythonmachine-learningscikit-learnmetricscross-validation
Ne-oL
Ne-oL

Reputation: 13

Scikit-Learn LOOCV vs doing it manually give different results, why?

So i have built a model for a small dataset and since it was a small dataset, i made a Leave-One-out Cross-Validation (LOOCV) check for its accuracy. so in short, i would remove one sample manually, train the model, predict the left out sample and save the prediction and repeat the process for all the samples. then i would use the list of predictions and the actual values to get a RMSE and R2. and today i found out that there was a Scikit-Learn implementation sklearn.model_selection.LeaveOneOut, however, when i tried it, it gave me different results for the RMSE, and refused to use R-squared as accuracy in the LOOCV method (it seems to calculate the accuracy per sample which does not work with R2).

here is a brief example of the code:

from numpy import mean
from numpy import std
from sklearn.datasets import make_blobs
from sklearn.model_selection import LeaveOneOut
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

cv = LeaveOneOut()
model = RandomForestRegressor(n_estimators=200, max_depth=6,n_jobs=40, random_state=0)

scores = cross_val_score(model, data2SN, labelCL, scoring='neg_root_mean_squared_error', cv=cv, n_jobs=-1)
# report performance
print('Accuracy: %.3f (%.3f)' % (mean(scores), std(scores)))

my guess is that I'm calculating the RMSE for the whole dataset, while the LOOCV is doing it per sample and eventually i would take the mean and this is what causes the discrepancy between the two codes output, however, when i tried to calculate the RMSE per sample it failed (citing this TypeError: Singleton array 3021.0 cannot be considered a valid collection). so I'm not sure how the RMSE is calculated inside the LOOCV. and I'm not sure to trust my code or just blindly use scikit-learn implementation.
I'm lost at what to do and chatGPT was just confusing as hell, so my human brethren please help

Upvotes: 0

Views: 823

Answers (2)

Emerson Vero
Emerson Vero

Reputation: 11

My intuition is that combining loocv and rmse can be misleading, because you will have one curve computed using rmse (training) and the other with MAE (validation). So, you lose interpretability between the two curves.

This would not happen if you use mean squared error instead of root mean squared error.

Upvotes: 0

Ben Reiniger
Ben Reiniger

Reputation: 12698

cross_val_score average the scores across folds, so with cv=LeaveOneOut() yes, it's computing the score per row (by a model trained on all other rows). With RMSE, that's equivalent to MAE; and R2 will just fail.

You could use cross_val_predict to get the individual predictions, then score that collection all at once, to reproduce your manual work.

Upvotes: 0

Related Questions