Aggregate over categories that contain NAs with ddply and lapply?

Question

I would like to aggregate a data.frame over 3 categories, with one of them varying. Unfortunately this one varying category contains NAs (actually it's the reason why it needs to vary). Thus I created a list of data.frames. Every data.frame within this list contains only complete cases with respect to three variables (with only one of them changing).

Let's reproduce this:

library(plyr)

mydata <- warpbreaks
names(mydata) <- c("someValue","group","size")
mydata$category <- c(1,2,3)
mydata$categoryA <- c("A","A","X","X","Z","Z")
# add some NA
mydata$category[c(8,10,19)] <- NA
mydata$categoryA[c(14,1,20)] <- NA 

# create a list of dfs that contains TRUE FALSE
noNAList <- function(vec){
res <- !is.na(vec)
return(res)
}

testTF <- lapply(mydata[,c("category","categoryA")],noNAList)

# create a list of data.frames
selectDF <- function(TFvec){
res <- mydata[TFvec,]
return(res)
}

# check x and see that it may contain NAs as long
# as it's not in one of the 3 categories I want to aggregate over    
x <-lapply(testTF,selectDF)

## let's ddply get to work
doddply <- function(df){
ddply(df,.(group,size),summarize,sumTest = sum(someValue))
}

y <- lapply(x, doddply);y

y comes very close to what I want to get

$category
group size sumTest
1     A    L     375
2     A    M     198
3     A    H     185
4     B    L     254
5     B    M     259
6     B    H     169

$categoryA
group size sumTest
1     A    L     375
2     A    M     204
3     A    H     200
4     B    L     254
5     B    M     259
6     B    H     169

But I need to implement aggregation over a third varying variable, which is in this case category and categoryA. Just like:

group size category sumTest sumTestTotal      
1      A    H        1      46          221 
2      A    H        2      46          221 
3      A    H        3      93          221 

and so forth. How can I add names(x) to lapply, or do I need a loop or environment here?

EDIT: Note that I want EITHER category OR categoryA added to the mix. In reality I have about 15 mutually exclusive categorical vars.

Answer 1

Finally, I found a solution that might not be as slick as Josh' but it works without no dark forces (data.table). You may laugh – here's my reproducible example using the same sample data as in the question.

qual <- c("category","categoryA")

# get T / F vectors
noNAList <- function(vec){
res <- !is.na(vec)
return(res)
}

selectDF <- function(TFvec) mydata[TFvec,]

NAcheck <- lapply(mydata[,qual],noNAList)

# create a list of data.frames
listOfDf <- lapply(NAcheck,selectDF)

workhorse <- function(charVec,listOfDf){
dfs <- list2env(listOfDf)
# create expression list
exlist <- list()
for(i in 1:length(qual)){
exlist[[qual[i]]] <- parse(text=paste("ddply(",qual[i],
                                  ",.(group,size,",qual[i],"),summarize,sumTest =    sum(someValue))",
                                  sep=""))
}

res <- lapply(exlist,eval,envir=dfs)
return(res)

}

Answer 2

Is this more like what you mean? I find your example extremely difficult to understand. In the below code, the method can take any column, and then aggregate by it. It can return multiple aggregation functions of someValue. I then find all the column names you would like to aggregate by, and then apply the function to that vector.

# Build a method to aggregate by column.
agg.by.col = function (column) {
    by.list=list(mydata$group,mydata$size,mydata[,column])
    names(by.list) = c('group','size',column)
    aggregate(mydata$someValue, by=by.list, function(x) c(sum=sum(x),mean=mean(x)))
}

# Find all the column names you want to aggregate by
cols = names(mydata)[!(names(mydata) %in% c('someValue','group','size'))]

# Apply the method to each column name.
lapply (cols, agg.by.col)

Answer 3

I know the question explicitly requests a ddply()/lapply() solution.

But ... if you are willing to come on over to the dark side, here is a data.table()-based function that should do the trick:

# Convert mydata to a data.table
library(data.table)
dt <- data.table(mydata, key = c("group", "size"))

# Define workhorse function
myfunction <- function(dt, VAR) {
    E <- as.name(substitute(VAR))
    dt[i = !is.na(eval(E)), 
       j = {n <- sum(.SD[,someValue]) 
            .SD[, list(sumTest = sum(someValue),
                       sumTestTotal = n,
                       share = sum(someValue)/n), 
                by = VAR]
           }, 
       by = key(dt)]
}

# Test it out
s1 <- myfunction(dt, "category")
s2 <- myfunction(dt, "categoryA")

ADDED ON EDIT

Here's how you could run this for a vector of different categorical variables:

catVars <- c("category", "categoryA")

ll <- lapply(catVars, 
             FUN = function(X) {
                       do.call(myfunction, list(dt, X))
                   })
names(ll) <- catVars

lapply(ll, head, 3)
# $category
#      group size category sumTest sumTestTotal     share
# [1,]     A    H        2      46          185 0.2486486
# [2,]     A    H        3      93          185 0.5027027
# [3,]     A    H        1      46          185 0.2486486
# 
# $categoryA
#      group size categoryA sumTest sumTestTotal share
# [1,]     A    H         A      79          200 0.395
# [2,]     A    H         X      68          200 0.340
# [3,]     A    H         Z      53          200 0.265

Answer 4

I think you might be making this really hard on yourself, if I understand your question correctly.

If you want to aggregate the data.frame 'myData' by three (or four) variables, you would simply do this:

aggregate(someValue ~ group + size + category + categoryA, sum, data=mydata)

   group size category categoryA someValue
1      A    L        1         A        51
2      B    L        1         A        19
3      A    M        1         A        17
4      B    M        1         A        63

aggregate will automatically remove rows that include NA in any of the categories. If someValue is sometimes NA, then you can add the parameter na.rm=T.

I also noted that you put a lot of unnecessary code into functions. For example:

# create a list of data.frames
selectDF <- function(TFvec){
    res <- mydata[TFvec,]
    return(res)
}

Can be written like:

selectDF <- function(TFvec) mydata[TFvec,] 

Also, using lapply to create a list of two data frames without the NA is overkill. Try this code:

x = list(mydata[!is.na(mydata$category),],mydata[!is.na(mydata$categoryA),])