How to visualize a Regression Tree in Python

Question

I'm looking to visualize a regression tree built using any of the ensemble methods in scikit learn (gradientboosting regressor, random forest regressor,bagging regressor). I've looked at this question which comes close, and this question which deals with classifier trees. But these questions require the 'tree' method, which is not available to the regression models in SKLearn.

but it didn't seem to yield a result. I'm running into issues because there is no .tree method for the regression versions of these trees (the method only exists for the classification versions). I'd like an output resembling this but based on a sci kit learn-constructed tree.

I've explored the methods associated with the objects but just cannot produce an answer.

Answer 1

A simple text mode solution:

from sklearn import tree
print(tree.export_text(clf_model))

typical result:

|--- feature_3 <= 0.46
|   |--- feature_0 <= 0.50
|   |   |--- feature_7 <= 0.50
|   |   |   |--- feature_5 <= 0.78
|   |   |   |   |--- value: [3288.64]
etc...

use clf_model.feature_names_in_ to get feature names

credits to https://mljar.com/blog/visualize-decision-tree/

Answer 2

DTR will sort of create a partition level for all the values
Check the graph - Click here

from sklearn.tree import DecisionTreeRegressor

#Getting X and y variable
X = df.iloc[:,1:2].values
y =df.iloc[:,2].values

#Creating a model object and fiting the data
reg = DecisionTreeRegressor(random_state=0)
reg.fit(X,y)

# Visualising the Decision Tree Regression results (higher resolution)
X_grid = np.arange(min(X), max(X), 0.01)
X_grid = X_grid.reshape((len(X_grid), 1))
plt.scatter(X, y, color = 'red')
plt.plot(X_grid, reg.predict(X_grid), color = 'blue')
plt.title('Truth or Bluff (Decision Tree Regression)')
plt.xlabel('Position level')
plt.ylabel('Salary')
plt.show()

Answer 3

As I commented, there is no functional difference between a classification and a regression decision tree plot. Adapting the regression toy example from the docs:

from sklearn import tree
X = [[0, 0], [2, 2]]
y = [0.5, 2.5]
clf = tree.DecisionTreeRegressor()
clf = clf.fit(X, y)

and then, similarly, some code from the classification docs regarding graphviz:

import graphviz 
dot_data = tree.export_graphviz(clf, out_file='tree.dot') 

we end up with a file tree.dot, looking like that:

digraph Tree {
node [shape=box] ;
0 [label="X[0] <= 1.0\nmse = 1.0\nsamples = 2\nvalue = 1.5"] ;
1 [label="mse = 0.0\nsamples = 1\nvalue = 0.5"] ;
0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ;
2 [label="mse = 0.0\nsamples = 1\nvalue = 2.5"] ;
0 -> 2 [labeldistance=2.5, labelangle=-45, headlabel="False"] ;
}

Now, you can proceed to visualize it as shown in the docs - but if for whatever reason you cannot render Graphviz objects, you can use the handy service WebGraphviz (+1 to the relevant answer in the linked question); the result looks like this:

Your own answer, i.e. going all the way to install graphlab just for the visualization, sounds like overkill...

Last remark: don't get deceived by the superficial differences in the tree layouts, which reflect only design choices of the respective visualization packages; the regression tree you have plotted (which, admittedly, does not look much like a tree) is structurally similar to the classification one taken from the docs - simply imagine a top-down tree, with your odor node at the top, followed by your green nodes and terminating to your blue & orange nodes (and replace "yes/no" with "True/False")...

Answer 4

After much searching, I found software offered by Turi that models a regression tree, not to be confused with a decision tree. Hope this helps

For what its worth, a regression tree looks like this:

While a decision/classifier tree looks like this:

And tho they look the same, the attribute needed to create this is tree_ which is only available to classifiers, not regressors.