Troubleshooting ddply() script

Question

I am developing a censored dependent variable for use in survival analysis. My goal is to find the last time ("time") that someone answers a question in a survey (e.g. the point where "q.time" is coded as "1", and "q.time+1" and q at all subsequent times are coded as "0").

By this logic, the last question answered should be coded as "1" (q.time). The first question that is NOT answered (q.time+1) should be coded as "0". And all questions subsequent to the first question NOT answered should be coded as "NA". I then want to remove ALL rows where the DV=NA from my dataset.

A very generous coworker has helped me to develop the following code, but he's on vacation now and it needs a little more lovin'. Code is as follows:

library(plyr)  # for ddply 
library(stats)  # for reshape(...) 
# From above 
dat <- data.frame( 
  id=c(1, 2, 3, 4), 
  q.1=c(1, 1, 0, 0), 
  q.2=c(1, 0, 1, 0), 
  dv.1=c(1, 1, 1, 1), 
  dv.2=c(1, 1, 0, 1)) 
# From above 
  long <- reshape(dat, 
                direction='long', 
                varying=c('q.1', 'q.2', 'dv.1', 'dv.2')) 
   ddply(long, .(id), function(df) { 
# figure out the dropoff time 
answered <- subset(df, q == 1) 
last.q = max(answered$time) 
subs <- subset(df, time <= last.q + 1) 
# set all the dv as desired 
new.dv <- rep(last.q,1) 
if (last.q < max(df$time)) new.dv <- c(0,last.q) 
subs$dv <- new.dv 
subs 
})

Unfortunately, this yields the error message:

"Error in `$<-.data.frame`(`*tmp*`, "dv", value = c(0, -Inf)) : 
 replacement has 2 rows, data has 0"

Any ideas? The problem seems to be located in the "rep" command, but I'm a newbie to R. Thank you so much!

UPDATE: SEE EXPLANATIONS BELOW, and then REFER TO FOLLOW-UP QUESTION

Hi there-I completely followed you, and really appreciate the time you took to help me out. I went back into my data and coded in a dummy Q where all respondents have a value of "1" - but, discovered where the error really may be. In my real data set, I have 30 questions (i.e., 30 times in long form). After I altered the dataset so FOR SURE q==1 for all id variables, the error message changed to saying

"Error in `$<-.data.frame`(`*tmp*`, "newvar", value = c(0, 29)) : replacement has 2 rows, data has 31"

If the problem is with the number of rows assigned to subs, then is the source of the error coming from...

subs <- subset(df, time <= last.q + 1) 

i.e., $time <= last.q + 1$ is setting the number of rows to the value EQUAL to last.q+1?

UPDATE 2: What, ideally, I'd like my new variable to look like!

 id  time q  dv   
 1    1   1   1
 1    2   1   1
 1    3   1   1
 1    4   1   1
 1    5   0   0
 1    6   0   NA
 2    1   1   1
 2    2   1   1
 2    3   0   0
 2    4   0   NA
 2    5   0   NA
 2    6   0   NA

Please note that "q" can vary between "0" or "1" over time (See the observation for id=1 at time=2), but due to the nature of survival analysis, "dv" cannot. What I need to do is create a variable that finds the LAST time that "q" changes between "1" and "0", and then is censored accordingly. After Step 4, my data should look like this:

 id  time q  dv   
 1    1   1   1
 1    2   1   1
 1    3   1   1
 1    4   1   1
 2    1   1   1
 2    2   1   1
 2    3   0   0

Answer 1

First, to give credit where credit is due, the code below is not mine. It was generated in collaboration with another very generous coworker (and engineer) who helped me work through my problem (for hours!).

I thought that other analysts tasked with constructing a censored variable from survey data might find this code useful, so I am passing the solution along.

library(plyr)
#A function that only selects cases before the last time "q" was coded as "1"
slicedf <- function(df.orig, df=NULL) {
if (is.null(df)) {
    return(slicedf(df.orig, df.orig))
}
if (nrow(df) == 0) {
    return(df)
}
target <- tail(df, n=1)
   #print(df)
   #print('--------')
   if (target$q == 0) {
       return(slicedf(df.orig, df[1:nrow(df) - 1, ]))
   }
if (nrow(df.orig) == nrow(df)) {
    return(df.orig)
}
return(df.orig[1:(nrow(df) + 1), ])
}
#Applies function to the dataset, and codes over any "0's" before the last "1" as "1"
long <- ddply(long, .(id), function(df) {
df <- slicedf(df)
if(nrow(df) == 0) {
return(df)
}
q <- df$q
if (tail(q, n=1) == 1) {
df$q <- rep(1, length(q))
} else {
df$q <- c(rep(1, length(q) - 1), 0)
}
return(df)
})

Thanks to everyone online who commented for your patience and help.

Answer 2

In short: The error is because there is no q == 1 when id == 4.

---

A good way to check what's going on here is to rewrite the function separately, and manually test each chunk that ddply is processing.

So first rewrite your code like this:

myfun <- function(df) { 
  # figure out the dropoff time 
  answered <- subset(df, q == 1) 
  last.q = max(answered$time) 
  subs <- subset(df, time <= last.q + 1) 
  # set all the dv as desired 
  new.dv <- rep(last.q,1) 
  if (last.q < max(df$time)) new.dv <- c(0,last.q) 
  subs$dv <- new.dv 
  subs
}
ddply(long, .(id), myfun )

That still gives an error of course, but at least now we can manually check what ddply is doing.

ddply(long, .(id), myfun ) really means:

1. Take the dataframe called long
2. Create a number of subset dataframes (one for each distinct id)
3. Apply the function myfun to each subsetted dataframe
4. Reassemble the results into a single dataframe

So let's attempt to do manually what ddply is doing automatically.

    > myfun(subset(long, id == 1))
        id time q dv
    1.1  1    1 1  2
    1.2  1    2 1  2
    > myfun(subset(long, id == 2))
        id time q dv
    2.1  2    1 1  0
    2.2  2    2 0  1
    > myfun(subset(long, id == 3))
        id time q dv
    3.1  3    1 0  2
    3.2  3    2 1  2
    > myfun(subset(long, id == 4))
    Error in `$<-.data.frame`(`*tmp*`, "dv", value = c(0, -Inf)) : 
      replacement has 2 rows, data has 0
    In addition: Warning message:
    In max(answered$time) : no non-missing arguments to max; returning -Inf
    > 

So it seems like the error is coming from the step where ddply applies the function for id == 4.

Now let's take the code outside of the function so we can examine each chunk.

> #################
> # set the problem chunk to "df" so we 
> # can examine what the function does
> # step by step
> ################
> df <- subset(long, id == 4)
> 
> ###################
> # run the bits of function separately
> ###################
> answered <- subset(df, q == 1) 
> answered
[1] id   time q    dv  
<0 rows> (or 0-length row.names)
> last.q = max(answered$time) 
Warning message:
In max(answered$time) : no non-missing arguments to max; returning -Inf
> last.q
[1] -Inf
> subs <- subset(df, time <= last.q + 1) 
> subs
[1] id   time q    dv  
<0 rows> (or 0-length row.names)
> # set all the dv as desired 
> new.dv <- rep(last.q,1) 
> new.dv
[1] -Inf
> if (last.q < max(df$time)) new.dv <- c(0,last.q)  
> subs$dv <- new.dv 
Error in `$<-.data.frame`(`*tmp*`, "dv", value = c(0, -Inf)) : 
  replacement has 2 rows, data has 0
> subs
[1] id   time q    dv  
<0 rows> (or 0-length row.names)
> 

So the error that you're getting comes from subs$dv <- new.dv because new.dv has length two (i.e. two values - (0, -Inf)) but sub$dv is length 0. That wouldn't be a problem if dv were a simple vector, but because it's in the sub dataframe whose columns all have two rows, then sub$dv must also have two rows.

The reason sub has zero rows is because there is no q == 1 when id == 4.

Should the final data frame not have anything for id == 4? The answer to your problem really depends on what you want to happen in the case when there are no q==1 for an id. Just let us know, and we can help you with the code.

UPDATE:

The error that you're getting is because subs$dv has 31 values in it and new.dv has two values in it.

In R when you try to assign a longer vector to a shorter vector, it will always complain.

> test <- data.frame(a=rnorm(100),b=rnorm(100))
> test$a <- rnorm(1000)
Error in `$<-.data.frame`(`*tmp*`, "a", value = c(-0.0507065994549323,  : 
  replacement has 1000 rows, data has 100
> 

But when you assign a shorter vector to a longer vector, it will only complain if the shorter is not an even multiple of the longer vector. (eg 3 does not go evenly into 100)

> test$a <- rnorm(3)
Error in `$<-.data.frame`(`*tmp*`, "a", value = c(-0.897908251650798,  : 
  replacement has 3 rows, data has 100

But if you tried this, it wouldn't complain since 2 goes into 100 evenly.

> test$a <- rnorm(2)
>

Try this:

 > length(test$a)
[1] 100
> length(rnorm(2))
[1] 2
> test$a <- rnorm(2)
> length(test$a)
[1] 100
>

What's it's doing is silently repeating the shorter vector to fill up the longer vector.

And again, what you do to get around the error (i.e. make both vectors the same length) will depend on what you're trying to achieve. Do you make new.dv shorter, or subs$dv longer?

Answer 3

.(id) in plyr is equivalent to

> dum<-split(long,long$id)
> dum[[4]]
    id time q dv
4.1  4    1 0  1
4.2  4    2 0  1

your problem is in your 4th split. You reference

answered <- subset(df, q == 1)

in your function. This is an empty set as there are no dum[[4]]$q taking value 1

If you just want to ignore this split then something like

ans<-ddply(long, .(id), function(df) { 
# figure out the dropoff time 
answered <- subset(df, q == 1) 
if(length(answered$q)==0){return()}
last.q = max(answered$time) 
subs <- subset(df, time <= last.q + 1) 
# set all the dv as desired 
new.dv <- rep(last.q,1) 
if (last.q < max(df$time)) new.dv <- c(0,last.q) 
subs$dv <- new.dv 
subs 
})

> ans
  id time q dv
1  1    1 1  2
2  1    2 1  2
3  2    1 1  0
4  2    2 0  1
5  3    1 0  2
6  3    2 1  2

would be the result