CODEWITHSUNDEEP
rsvmr-caret
Jack Armstrong
Jack Armstrong

Reputation: 1239

Linear SVM and extracting the weights

I am practicing SVM in R using the iris dataset and I want to get the feature weights/coefficients from my model, but I think I may have misinterpreted something given that my output gives me 32 support vectors. I was under the assumption I would get four given I have four variables being analyzed. I know there is a way to do it when using the svm() function, but I am trying to use the train() function from caret to produce my SVM. 

library(caret)

# Define fitControl
fitControl <- trainControl(## 5-fold CV
              method = "cv",
              number = 5,
              classProbs = TRUE,
              summaryFunction = twoClassSummary )

# Define Tune
grid<-expand.grid(C=c(2^-5,2^-3,2^-1))

########## 
df<-iris head(df)
df<-df[df$Species!='setosa',]
df$Species<-as.character(df$Species)
df$Species<-as.factor(df$Species)

# set random seed and run the model
set.seed(321)
svmFit1 <- train(x = df[-5],
                 y=df$Species,
                 method = "svmLinear", 
                 trControl = fitControl,
                 preProc = c("center","scale"),
                 metric="ROC",
                 tuneGrid=grid )
svmFit1

I thought it was simply svmFit1$finalModel@coefbut I get 32 vectors when I believe I should get 4. Why is that?

Upvotes: 2

Views: 2086

Answers (2)

Benjamin Phua
Benjamin Phua

Reputation: 53

As more folks start moving from Caret to Tidymodels I thought I'd put a version of the above solution for Tidymodels Aug 2020 because I don't see many discussions about this so far and it isn't that straightforward to do.

Outlining the main steps here but please review the links at the end for detail for why it was done this way.

1. Get Your Final Model

set.seed(2020)

# Assuming kernlab linear SVM

# Grid Search Parameters
tune_rs <- tune_grid(
  model_wf,
  train_folds,
  grid = param_grid,
  metrics = classification_measure,
  control = control_grid(save_pred = TRUE)
)

# Finalise workflow with the parameters for best accuracy
best_accuracy <- select_best(tune_rs, "accuracy")

svm_wf_final <- finalize_workflow(
  model_wf,
  best_accuracy
)

# Fit on your final model on all available data at the end of experiment
final_model <- fit(svm_wf_final, data)
# fit takes a model spec and executes the model fit routine (Parsnip)
  # model_spec, formula and data to fit upon

2. Extract the KSVM Object, Pull Required Info, Calculate Variable Importance

ksvm_obj <- pull_workflow_fit(final_model)$fit
# Pull_workflow_fit returns the parsnip model fit object
# $fit returns the object produced by the fitting fn (which is what we need! and is dependent on the engine)

coefs <- ksvm_obj@coef[[1]]
# first bit of info we need are the coefficients from the linear fit

mat <- ksvm_obj@xmatrix[[1]]
# xmatrix that we need to matrix multiply against

var_impt <- coefs %*% mat
# var importance

Ref:

  1. Extracting the Weights of Support Vectors using Caret: Linear SVM and extracting the weights

  2. Variable Importance (Last Section of this post): http://www.rebeccabarter.com/blog/2020-03-25_machine_learning/#finalize-the-workflow

Upvotes: 1

tmastny
tmastny

Reputation: 417

So coef is not the weight W of the support vectors. Here's the relevant section of the ksvm class in the docs:

coef The corresponding coefficients times the training labels.

To get what you are looking for, you'll need to do the following:

coefs <- svmFit1$finalModel@coef[[1]]
mat <- svmFit1$finalModel@xmatrix[[1]]

coefs %*% mat

See below for a reproducible example.

library(caret)
#> Loading required package: lattice
#> Loading required package: ggplot2
#> Warning: package 'ggplot2' was built under R version 3.5.2

# Define fitControl
fitControl <- trainControl(
  method = "cv",
  number = 5,
  classProbs = TRUE,
  summaryFunction = twoClassSummary
)

# Define Tune
grid <- expand.grid(C = c(2^-5, 2^-3, 2^-1))

########## 
df <- iris 

df<-df[df$Species != 'setosa', ]
df$Species <- as.character(df$Species)
df$Species <- as.factor(df$Species)

# set random seed and run the model
set.seed(321)
svmFit1 <- train(x = df[-5],
                 y=df$Species,
                 method = "svmLinear", 
                 trControl = fitControl,
                 preProc = c("center","scale"),
                 metric="ROC",
                 tuneGrid=grid )

coefs <- svmFit1$finalModel@coef[[1]]
mat <- svmFit1$finalModel@xmatrix[[1]]

coefs %*% mat
#>      Sepal.Length Sepal.Width Petal.Length Petal.Width
#> [1,]   -0.1338791  -0.2726322    0.9497457    1.027411

Created on 2019-06-11 by the reprex package (v0.2.1.9000)

Sources

  1. https://www.researchgate.net/post/How_can_I_find_the_w_coefficients_of_SVM

  2. http://r.789695.n4.nabble.com/SVM-coefficients-td903591.html

  3. https://stackoverflow.com/a/1901200/6637133

Upvotes: 7

Related Questions