CODEWITHSUNDEEP
rdata.tablemissing-data
sds
sds

Reputation: 60004

How do I detect and re-insert missing data?

I have a missing row in a data table which describes a function from time, sid, and s.c to count:

> dates.dt[1001:1011]
        sid   s.c  count                time
 1: missing CLICK 104192 2013-05-25 10:00:00
 2: missing SHARE   7694 2013-05-25 10:00:00
 3: present CLICK  99573 2013-05-25 10:00:00
 4: present SHARE  89302 2013-05-25 10:00:00
 5: missing CLICK     28 2013-05-25 11:00:00
 6: present CLICK     25 2013-05-25 11:00:00
 7: present SHARE     15 2013-05-25 11:00:00
 8: missing CLICK 104544 2013-05-25 12:00:00
 9: missing SHARE   7253 2013-05-25 12:00:00
10: present CLICK 105891 2013-05-25 12:00:00
11: present SHARE  88709 2013-05-25 12:00:00

the missing row is (I expect a row for each of the two values of the 1st and 2nd columns and each time slice):

    missing SHARE      0 2013-05-25 11:00:00

How do I detect and restore such missing rows?

The way I discovered this was

library(data.table)
total <- dates.dt[, list(sum(count)) , keyby="time"]
setnames(total,"V1","total")
ts <- dates.dt[s.c=="SHARE" & sid=="missing", list(sum(count)) , keyby="time"]
cat("SHARE/missing:",nrow(ts),"rows\n")
stopifnot(identical(total$time,ts$time)) # --> ERROR!
total$shares.missing <- ts$V1

Now, I guess I can find the first place where ts$time and total$time differ and insert a 0 row there, but this seems like a rather tedious process.

Thanks!

Upvotes: 1

Views: 165

Answers (1)

eddi
eddi

Reputation: 49448

Following @Frank's suggestion you can do:

setkey(dt, time, sid, s.c)
dt[J(expand.grid(unique(time),unique(sid),unique(s.c)))][order(time, sid, s.c)]
#                   time     sid   s.c  count
# 1: 2013-05-25 10:00:00 missing CLICK 104192
# 2: 2013-05-25 10:00:00 missing SHARE   7694
# 3: 2013-05-25 10:00:00 present CLICK  99573
# 4: 2013-05-25 10:00:00 present SHARE  89302
# 5: 2013-05-25 11:00:00 missing CLICK     28
# 6: 2013-05-25 11:00:00 missing SHARE     NA
# 7: 2013-05-25 11:00:00 present CLICK     25
# 8: 2013-05-25 11:00:00 present SHARE     15
# 9: 2013-05-25 12:00:00 missing CLICK 104544
#10: 2013-05-25 12:00:00 missing SHARE   7253
#11: 2013-05-25 12:00:00 present CLICK 105891
#12: 2013-05-25 12:00:00 present SHARE  88709

Upvotes: 2

Related Questions