CODEWITHSUNDEEP
rbenchmarkingmlr3
ds_col
ds_col

Reputation: 139

How can I account for "failed" learners in benchmark()

I have a large list of task/learner/resampling combinations. I execute the resampling via

design = data.table(
  task = list_of_tasks,
  learner = list_of_learners,
  resampling = list_of_resamplings
)

bmr = benchmark(design) 

tab = bmr$aggregate(c(msr("classif.acc")))

The last command fails and I get the following error message:

Error in assert_classif(truth, response = response) : Assertion on 'response' failed: Contains missing values (element 1).

How can I check what went wrong? The learners have worked for slightly different tasks and they are all combination of "standard" learners (svm, naive bayes) with a preceding po("scale"). There are not missing data in the predictors of the targets of the tasks.

Upvotes: 0

Views: 125

Answers (1)

be-marc
be-marc

Reputation: 1491

At least one learner predicted NAs. Search for NAs in the predictions to identify the failing learner.

library(mlr3)
library(mlr3misc)

# experiment
learner_rpart = lrn("classif.rpart")
learner_debug = lrn("classif.debug", predict_missing = 0.5)
task = tsk("pima")
resampling = rsmp("cv", folds = 3)
design = benchmark_grid(task, list(learner_rpart, learner_debug), resampling)
bmr = benchmark(design)

# search for predictions with NAs
tab = as.data.table(bmr)
tab[map_lgl(tab$prediction, function(pred) any(is.na(pred$response)))]

You should post a new question with a reprex including the failing learner, task and resampling.

Upvotes: 1

Related Questions