CODEWITHSUNDEEP
rstatisticsdata.tablesummaryquantile
G. Bartowski
G. Bartowski

Reputation: 115

Look for various quantiles in a function list in an efficient way?

I have tried calling quantiles multiple times, but that is super inefficient considering that I need a quantile increment of 0.05 and my data is very big.

ff <- function(table, IDs, q.increment=0.05){
  fun <- list("mean" = function(x) mean(x, na.rm = TRUE),
              "median" = function(x) median(x, na.rm =TRUE),
              function(x) quantile(x, probs = seq(0, 1, q.increment), na.rm=T))

  fun.names <- names(fun)
  abc <- table[ , c(summary = list(fun.names),
                    lapply(.SD, function(col) I(lapply(fun, function(f) f(col))))),
                by = IDs, .SDcols = grep("value", names(table))]
  return(abc)
}

dt <- data.table(country = c("FR", "US", "HU", "HU", "FR", "FR", "US", "US", "US", "HU"), value1=rnorm(10), value2=rnorm(10))
abc <- ff(dt, c("country"))

What I dont like is that the quantiles go all in one cell, I would like to have a single row for each quantile.

Upvotes: 1

Views: 40

Answers (1)

akrun
akrun

Reputation: 887213

The way to do this would be to convert the quantile to list and then while unlisting use recursive = FALSE

dt[, c(unlist(lapply(.SD, function(x) list(mean = mean(x),
    median = median(x))), recursive = FALSE), 
   unlist(lapply(.SD, function(x) as.list(quantile(x,
   probs = seq(0, 1, 0.05)))), recursive = FALSE)), country]

It can be wrapped in a function

ff1 <- function(data, IDs, q.increment = 0.05) {
    f1 <- function(x) list(mean = mean(x, na.rm = TRUE),
                           median = median(x, na.rm = TRUE),
                           quantile = as.list(quantile(x, 
                probs = seq(0, 1, q.increment))))


     data[, unlist(unlist(lapply(.SD, f1), recursive = FALSE),
           recursive = FALSE), by = IDs, .SDcols = grep("value", names(data))]
 }

out <- ff1(dt, "country")

If we need it in a long format, then use melt

nm1 <- unique(sub(".*\\.", "", names(out)[-1]))
melt(out, measure = patterns('^value1', '^value2'),
      variable.name = 'summary')[, summary := nm1[summary]][]
# country summary      value1      value2
# 1:      FR    mean -0.70362861 -0.37004727
# 2:      US    mean -0.17024421 -0.10986835
# 3:      HU    mean  0.35754440  0.43067053
# 4:      FR  median -0.25453398 -0.72539656
# 5:      US  median -0.08068703  0.15472558
# 6:      HU  median  0.61732639  0.30846369
# 7:      FR      0% -1.60473855 -1.34258692
# 8:      US      0% -0.87641285 -2.04386860
# 9:      HU      0% -0.37871048  0.08147549
#10:      FR      5% -1.46971809 -1.28086789
#11:      US      5% -0.80765939 -1.72964937
#...

Upvotes: 1

Related Questions