CODEWITHSUNDEEP
rdata.table
alexvpickering
alexvpickering

Reputation: 642

Group by one column, select row with minimum in one column for every pair of columns

Difficult question to phrase. Here is an example of what I would like to do. An example of what I am starting with:

set.seed(0)
dt <- data.table(dr1.d=rnorm(5), dr1.p=abs(rnorm(5, sd=0.08)),
                 dr2.d=rnorm(5), dr2.p=abs(rnorm(5, sd=0.08)),
                 dr3.d=rnorm(5), dr3.p=abs(rnorm(5, sd=0.08)),
                 dr4.d=rnorm(5), dr4.p=abs(rnorm(5, sd=0.08)),
                 sym = paste("sym", c(1,1,1,2,2)))
dt

      dr1.d        dr1.p      dr2.d      dr2.p       dr3.d       dr3.p      dr4.d      dr4.p   sym
1:  1.2629543 0.1231960034  0.7635935 0.03292087 -0.22426789 0.040288638 -0.2357066 0.09215294 sym 1
2: -0.3262334 0.0742853628 -0.7990092 0.02017788  0.37739565 0.086861549 -0.5428883 0.07937283 sym 1
3:  1.3297993 0.0235776357 -1.1476570 0.07135369  0.13333636 0.055276307 -0.4333103 0.03436105 sym 1
4:  1.2724293 0.0004613738 -0.2894616 0.03485466  0.80418951 0.102767948 -0.6494716 0.09906433 sym 2
5:  0.4146414 0.1923722711 -0.2992151 0.09900307 -0.05710677 0.003738094  0.7267507 0.02234770 sym 2

For all pairs of columns that share a drug (e.g. 'dr1') I want to group rows by 'sym', then select the row with the smallest p-value (ends in '.p') within each group. The final result of the above data.table would be this: 

       dr1.d        dr1.p      dr2.d      dr2.p       dr3.d       dr3.p      dr4.d      dr4.p   sym
1:  1.3297993 0.0235776357  -0.7990092 0.02017788 -0.22426789 0.040288638 -0.4333103 0.03436105 sym 1
2:  1.2724293 0.0004613738  -0.2894616 0.03485466 -0.05710677 0.003738094  0.7267507 0.02234770 sym 2

I have tried using .SD and lapply to accomplish this, but I can't wrap my head around it. Thank you!

Upvotes: 9

Views: 237

Answers (3)

Arun
Arun

Reputation: 118889

The most important (and powerful) thing to understand about data.table is that, as long as j returns a list, each element of the list will become a column in the result.

With that knowledge and some base R fun, we can get this result directly by doing:

# I'm on v1.9.7, see https://github.com/Rdatatable/data.table/wiki/Installation
cols1 = grep("d$", names(dt), value=TRUE)
cols2 = grep("p$", names(dt), value=TRUE)
dt[, Map(`[`, mget(c(cols1,cols2)), lapply(mget(cols2), which.min)), by=sym]
#      sym    dr1.d      dr2.d       dr3.d      dr4.d        dr1.p      dr2.p
# 1: sym 1 1.329799 -0.7990092 -0.22426789 -0.4333103 0.0235776357 0.02017788
# 2: sym 2 1.272429 -0.2894616 -0.05710677  0.7267507 0.0004613738 0.03485466
#          dr3.p      dr4.p
# 1: 0.040288638 0.03436105
# 2: 0.003738094 0.02234770

See the vignettes for more.

Upvotes: 13

talat
talat

Reputation: 70336

Here's an all-in-one approach, though you might want to split it into separate steps for readability:

dcast(melt(dt, measure = patterns("\\.p$", "\\.d$"), id.vars = "sym", 
  value.name = c("p", "d"))[, .SD[which.min(p)], by = list(sym, variable)], 
  sym ~ variable, value.var = c("p", "d"))
#     sym          p_1        p_2         p_3        p_4      d_1        d_2         d_3        d_4
#1: sym 1 0.0235776357 0.02017788 0.040288638 0.03436105 1.329799 -0.7990092 -0.22426789 -0.4333103
#2: sym 2 0.0004613738 0.03485466 0.003738094 0.02234770 1.272429 -0.2894616 -0.05710677  0.7267507

It's essentially melting by two patterns first, then subsetting by minimum p-value, then dcasting back to wide format.

Upvotes: 3

Chris
Chris

Reputation: 6372

With some melting and casting this is fairly straightforward

library(data.table)

set.seed(0)
dt <- data.table(dr1.d=rnorm(5), dr1.p=abs(rnorm(5, sd=0.08)),
                 dr2.d=rnorm(5), dr2.p=abs(rnorm(5, sd=0.08)),
                 dr3.d=rnorm(5), dr3.p=abs(rnorm(5, sd=0.08)),
                 dr4.d=rnorm(5), dr4.p=abs(rnorm(5, sd=0.08)),
                 sym = paste("sym", c(1,1,1,2,2)))


dt[, rowid := .I] #add a row identifier
dt <- melt(dt, id.vars = c("sym", "rowid"), variable.factor = F)

dt[, c("col","val") := tstrsplit(variable, "." , fixed = T)] #split the column so we can group
dt[, variable := NULL] #small cleanup


dt <- dcast(dt, sym + rowid + col ~ val)
dt <- dt[, .SD[which.min(p)], by = .(sym,col)] #select min row

dt[, rowid := NULL] #cleanup

dt <- dcast(melt(dt, id.vars = c("sym","col")), sym ~ col + variable)
dt
         sym    dr1_d        dr1_p      dr2_d      dr2_p       dr3_d       dr3_p      dr4_d      dr4_p
1: sym 1 1.329799 0.0235776357 -0.7990092 0.02017788 -0.22426789 0.040288638 -0.4333103 0.03436105
2: sym 2 1.272429 0.0004613738 -0.2894616 0.03485466 -0.05710677 0.003738094  0.7267507 0.02234770

Upvotes: 2

Related Questions