CODEWITHSUNDEEP
rimputation
Debasish
Debasish

Reputation: 63

Error in optim: L-BFGS-B needs finite value of fn

I am trying to run impute_errors() function of the imputeTestBench package for a series of values. I am using six user defined methods for selection of best imputation method. Below is my code:

    correctedSalesHistoryMatrix[, 1:2],
    matrix(unlist(apply(X = as.matrix(correctedSalesHistoryMatrix[, -c(1, 2)]),
                        MARGIN = 1,
                        FUN = impute_errors,
                        smps = "mcar", 
                        methods = c(
                          "imputationMethod1"
                          , "imputationMethod2"
                          , "imputationMethod3"
                          , "imputationMethod4"
                          , "imputationMethod5"
                          , "imputationMethod6"
                        ), 
                        methodPath = "C:\\Documents\\Imputations.R",
                        errorParameter = "mape",
                        missPercentFrom = 10,
                        missPercentTo = 10
                        )
                  ), nrow = nrow(correctedSalesHistoryMatrix), byrow = T
           )
    )

When I am using a small dataset, the function executes successfully. When I am using a large dataset I am using the following error:

Error in optim(init[mask], getLike, method = "L-BFGS-B", lower = rep(0, : L-BFGS-B needs finite values of 'fn' Called from: optim(init[mask], getLike, method = "L-BFGS-B", lower = rep(0, np + 1L), upper = rep(Inf, np + 1L), control = optim.control)

Upvotes: 0

Views: 1533

Answers (1)

Steffen Moritz
Steffen Moritz

Reputation: 7730

I don't think this is an easy fix.

Error is probably not caused by imputeTestBench itself, but rather by one of your user defined imputation methods.

Run impute_errors like before and only add na_mean as method instead of your user defined methods (impute_errors(..., methods = 'na_mean') ) to see if this suggestion is true.

The error itself occurs quite often and has to do with stats::optim receiving inputs it can't deal with. Quite likely you are not using stats::optim in your user defined imputation methods (so you can't easily fix the input). More likely is that a package your are using is doing some calculations and then using stats::optim. Or even worse a package you are using is using another package, that is using stats::optim.

In the answers to this question you can see an explanation underlying problem. Overall seems to occur especially for large datasets, when the fn input parameter to stats::optim becomes Inf.

Here a some examples of the problem also occurring for different R packages and functions (which all use stats::optim somewhere internally): 1, 2, 3

Not too much you can do overall, if you don't want to go extremely deep into the underlying packages.

If you are using the imputeTS package for one of your user supplied imputation methods, in this Github Issue a workaround is proposed, which might help, if the error occurs within the na_kalman or na_seadec method.

Upvotes: 1

Related Questions