CODEWITHSUNDEEP
rneural-networkr-caretnnet
mmmmmmmmmm
mmmmmmmmmm

Reputation: 445

How do I evaluate multiple models using the CARET package?

I want to use the CARET and nnet packages in order to predict survival/death for the Titanic data set. I would like to fit 20 neural networks each with 1 hidden node, 2 hidden nodes, ... 20 hidden nodes. Typically the CARET package will select the best model on the basis of the training data, but I want to take each of the 20 networks and apply each of them to the test data. How can I save each of the models in order to test them against the test data set? Is there a method in the CARET package that can help?

Upvotes: 2

Views: 1118

Answers (1)

scribbles
scribbles

Reputation: 4349

Not directly, no, but it should be possible. You would need to modify thefit function to save them out to a file. Inside the fit function, you would know the tuning parameter value but not what resample that the model was build with.

Here is an example of how you would go about accomplishing this, which comes from an answer on a very similar question. 

# Copy all model structure info from existing model type
cust.mdl <- getModelInfo("rf", regex=FALSE)[[1]]

# Override fit function so that we can save the iteration
cust.mdl$fit <- function(x=x, y=y, wts=wts, param=param, lev=lev, last=last, classProbs=classProbs, ...) {
  # Dont save the final pass (dont train the final model across the entire training set)
  if(last == TRUE) return(NULL) 

  # Fit the model
  fit.obj <- getModelInfo("rf", regex=FALSE)[[1]]$fit(x, y, wts, param, lev, last, classProbs, ...)

  # Create an object with data to save and save it
  fit.data <- list(resample=rownames(x),
                   mdl=fit.obj,
                   #x, y, wts,
                   param=param, lev=lev, last=last, classProbs=classProbs, 
                   other=list(...))

  # Create a string representing the tuning params
  param.str <- paste(lapply(1:ncol(param), function(x) {
                     paste0(names(param)[x], param[1,x])
                    }), collapse="-")

  save(fit.data, file=paste0("rf_modeliter_", sample(1000:9999,1), "_", param.str, ".RData"))
  return (fit.obj)
}

Upvotes: 3

Related Questions