CODEWITHSUNDEEP
pythonmachine-learningscikit-learndecision-tree
Jibin Mathew
Jibin Mathew

Reputation: 5102

How to know the factor by which a feature affects a model's prediction

I have trained my model on a data set and i used decision trees to train my model and it has 3 output classes - Yes,Done and No , and I got to know the feature that are most decisive in making a decision by checking feature importance of the classifier. I am using python and sklearn as my ML library. Now that I have found the feature that is most decisive I would like to know how that feature contributes, in the sense that if the relation is positive such that if the feature value increases the it leads to Yes and if it is negative It leads to No and so on and I would also want to know the magnitude for the same.

I would like to know if there a solution to this and also would to know a solution that is independent of the algorithm of choice, Please try to provide solutions that are not specific to decision tree but rather general solution for all the algorithms.

If there is some way that would tell me like:

for feature x1 the relation is 0.8*x1^2 for feature x2 the relation is -0.4*x2

just so that I would be able to analyse the output depends based on input feature x1 ,x2 and so on

Is it possible to find out the whether a high value for particular feature to a certain class, or a low value for the feature.

Upvotes: 0

Views: 1760

Answers (2)

Craig
Craig

Reputation: 346

You can use Partial Dependency Plots (PDPs). scikit has a built-in PDP for their GBM - http://scikit-learn.org/stable/modules/ensemble.html#partial-dependence which was created in Friedman's Greedy Function Approximation Paper http://statweb.stanford.edu/~jhf/ftp/trebst.pdf pp26-28.

If you used scikit-learn GBM, use their PDP function. If you used another estimator, you can create your own PDP which is a few lines of code. PDPs and this method is algorithm agnostic as you asked. It just will not scale.

Logic

  1. Take your training data
  2. For the feature you are examining, get all unique values or some quantiles to reduce the time
  3. Take a unique value
  4. For the feature you are examining, in all observations, replace with the value from (3)
  5. Predict all training observations
  6. Get the mean of all predictions
  7. Plot the point (unique value, mean)
  8. Repeat 3-7 taking the next unique value until no more values

You now have a 1-way PDP. When the feature increases (X-axis), what on average happens to the prediction (y-axis). What is the magnitude of the change.

Taking the analysis further, you can fit a smooth curve or splines to the PDP which may help understand the relationship. As @Maxim said, there is not a perfect rule so you are looking for the trend here, trying to understand a relationship. We tend to run this for the most important features and/or features you are curious about.

The above scikit-learn reference has more examples.

For a Decision Tree, you can use the algorithmic short-cut as described by Friedman and implemented by scikit-learn. You need to walk the tree so the code is tied to the package and algorithm, hence it does not answer your question and I will not describe it. But it is on that scikit-learn page I referenced and in the paper.

def pdp_data(clf, X, col_index):
    X_copy = np.copy(X)

    results = {}

    results['x_values'] = np.sort(np.unique(X_copy[:, col_index]))
    results['y_values'] = []

    for value in results['x_values']:
        X_copy[:, col_index] = value
        y_predict = clf.predict_log_proba(X_copy)[:, 1]
        results['y_values'].append(np.mean(y_predict))

    return results

Edited to answer new part of question: For the addition to your question, you are looking for a linear model with coefficients. If you must interpret the model with linear coefficients, build a linear model.

Sometimes how you need to interpret the model guides what type of model you build.

Upvotes: 1

Maksim Khaitovich
Maksim Khaitovich

Reputation: 4792

In general - no. Decision trees work differently that that. For example it could have a rule under the hood that if feature X > 100 OR X < 10 and Y = 'some value' than answer is Yes, if 50 < X < 70 - answer is No etc. In the instance of decision tree you may want to visualize its results and analyse the rules. With RF model it is not possible, as far as I know, since you have a lot of trees working under the hood, each has independent decision rules.

Upvotes: 0

Related Questions