CODEWITHSUNDEEP
pythonmachine-learningxgboostsupervised-learning
Simone Romeo
Simone Romeo

Reputation: 59

XGBoost: softprob & softmax yield inconsistent results

I'm using a set of data with six columns, and I am trying to predict the target feature that can be either 0,1 with xgboost.

As I'm still learning to use xgboost, I'm using both softprob and softmax as objectives. When I print them together at the end, the results of softprob and softmax are not consistent (meaning that xgboost didn't choose the feature with the highest probability). I don't understand whether my code has a bug or I have false expectations towards how xgboost works. Can you help me figure it out?

import xgboost as xgb
x_train,x_test,y_train,y_test=train_test_split(x, y, test_size=0.33,shuffle=False)

#Predicting with softprob

train = xgb.DMatrix(x_train,label=y_train)
test = xgb.DMatrix(x_test,label=y_test)   
params= {
    "max_depth":4,
    "eta":0.3,
    "objective":"multi:softprob",
    "num_class":2
    }
epochs=10 
model=xgb.train(params,train,epochs)
predictions=model.predict(test)
dim1,dim2 = predictions.shape
predictions=np.reshape(predictions,(dim2,dim1))
y_test=y_test.to_numpy()
results = pd.DataFrame(y_test,columns=(["Actual"]))
results["0"] = predictions[0]
results["1"] = predictions[1]

#Predicting with softmax

train = xgb.DMatrix(x_train,label=y_train)
test = xgb.DMatrix(x_test,label=y_test)
params= {
    "max_depth":4,
    "eta":0.3,
    "objective":"multi:softmax",
    "num_class":2
    }
epochs=10  
model=xgb.train(params,train,epochs)
predictions=model.predict(test)
results["Prediction"] = predictions



print(results)

Print result:

     Actual         0         1  Prediction
0       1.0  0.718783  0.670569         0.0
1       1.0  0.281217  0.685960         1.0
2       1.0  0.389004  0.314040         1.0
3       0.0  0.610996  0.685960         0.0
4       1.0  0.473767  0.314040         1.0
..      ...       ...       ...         ...
456     1.0  0.504193  0.710020         0.0
457     1.0  0.495807  0.580242         0.0
458     1.0  0.253023  0.419758         1.0
459     1.0  0.746977  0.608333         0.0
460     1.0  0.329431  0.391667         0.0

[461 rows x 4 columns]

Upvotes: 0

Views: 1413

Answers (1)

nickyfot
nickyfot

Reputation: 2019

A more formal answer of what is discussed in the comments, when you retrain models you end up with different trainable parameters in each model. Even if you retrain with the same hyper-parameters there is a chance you'll end up with different output for some samples particularly the ones close to the decision boundary. You can check the accuracy of both models and chose the best performing one if you are looking to optimize accuracy (although they shouldn't be too different). If you need both the softprob output and a class prediction, it is probably best to implement softmax on the softprob output yourself eg. using scipy

Upvotes: 1

Related Questions