CODEWITHSUNDEEP
rmlr
pokyah
pokyah

Reputation: 163

How to make the benchmark function not to fail if a specific learner fails on a specific task?

I need to compare various learners on multiple tasks (> 100 000 hourly temperature records from weather stations) in a benchmark experiment.

Because one of my learners may sometimes fails (and it is acceptable) I need a solution (I guess with trycatch() or purrr::possibly()) so that mlr returns, instead of an error for the whole benchmark, NULL for for the incriminated learner of the task on which it failed.

This will later allow me to understand in which situations it fails

So far I have achieved this :

bmrs = tasks %>%
          purrr::map(possibly(~mlr::benchmark(.,
            learners = my_learners,
            resamplings = mlr::makeResampleDesc("LOO"),
            measures = rmse,
            keep.pred = TRUE,
            models = FALSE), NULL))

Notice that I map the function mlr::benchamrk() to each task rather than passing all the tasks at once. This is precisely because my aim is to get a NULL value when a specific learner failing on a specific task throws an error. But doing so will returns a NULL value for the whole benchmark of my learners on the current task instead of a NULL only for the incriminated learner.

I've succeed to achieve what I want with the function mlr::resample() passed to a purrr::map() where it iterates on each of the learners but then I don't have all the bmr convenient functions mlr::getBMR...() that I'll might need later to perform some benchmark post-analysis like the merge benchmark results : 

   resample_by_task = function(t) {
     learners %>% purrr::map(possibly(
       ~ mlr::resample(.,
         task = t,
         resampling = mlr::makeResampleDesc("LOO"),
         measures = rmse,
         keep.pred = TRUE,
         models = models), NULL))}

   bmrs = purrr::map(tasks, ~resample_by_task(.))

So, would you advice me to work with the mlr::benchmark function wrapped by a custom error catching system or to work with the mlr::resample an do some custom code to work with its results ?

Thanks in advance for your support

Upvotes: 2

Views: 81

Answers (1)

Lars Kotthoff
Lars Kotthoff

Reputation: 109242

You could set the on.learner.error mlr package option, see https://mlr.mlr-org.com/reference/configureMlr.html#arguments.

Upvotes: 2

Related Questions