Solving R fatal error when using loo with GAM

Question

I have several GAMs (from bmrs) that I'd like to perform a leave-one-out (LOO) cross-validation with to determine the best model. However, every time I use loo, I run into issues that culminate in the R fatal error message and I have to start all over again. It's rather annoying as there doesn't seem to a good way to troubleshoot.

Details:

R version 4.2.2
Rstudio version 2023.12.0.369
brms version 2.20.4

Example code:

set.seed(42)
df <- data.frame(y = rnorm(2000, 100, 20),
           x = rnorm(2000, 50, 20),
           a = sample(LETTERS[1:3], 2000, replace = TRUE)) 
m <- brm(y ~ s(x) + s(x, a, bs = "fs"),
       prior = c(prior(normal(0, 10), class = Intercept),
                 prior(normal(0, 1), class = b),
                 prior(normal(0, 1), class = sigma),
                 prior(normal(0, 1), class = sds)),
       family = gaussian(link = "identity"), data = df)
loo(m)

At this point with my data, I usually get this warning:

Warning message:
Found 1 observations with a pareto_k > 0.7 in model 'm'. 
It is recommended to set 'moment_match = TRUE' in order to 
perform moment matching for problematic observations.  

So I modify my code, and then I get the fatal error message:

loo(m, moment_match = TRUE)