CODEWITHSUNDEEP
rdata.tablesubsetdate-range
yingw
yingw

Reputation: 307

efficiently subsetting data.table multiple times

I have data in this format

> data = data.table(id = 1:10, date = seq(as.Date("2016-01-01"), by = 1, length = 10))
> data
    id       date
 1:  1 2016-01-01
 2:  2 2016-01-02
 3:  3 2016-01-03
 4:  4 2016-01-04
 5:  5 2016-01-05
 6:  6 2016-01-06
 7:  7 2016-01-07
 8:  8 2016-01-08
 9:  9 2016-01-09
10: 10 2016-01-10

I have another matrix which is the queries / subsets that I wish to preform.

> query = data.table(id = c(1,4,7), date_start = c("2016-01-01", "2016-01-01", "2016-01-01"), date_end = c("2016-01-04", "2016-01-02", "2016-01-03"))
> query
   id date_start   date_end
1:  1 2016-01-01 2016-01-04
2:  4 2016-01-01 2016-01-02
3:  7 2016-01-01 2016-01-03

I wish to do something like this:

subset(data, (id == query[1] & date > date_start[1] & date < date_end[1]) | 
       (id == query[2] & date > date_start[2] & date < date_end[2]) |
       (id == query[3] & date > date_start[3] & date < date_end[3]))

Is there a automatically generate the subset query without using a for-loop and rbinding the result.

Thanks

Upvotes: 2

Views: 230

Answers (3)

Arun
Arun

Reputation: 118779

In the current development version, you can directly perform a non-equi join as follows:

# data.table v1.9.7+
data[query, .(id, x.date), on=.(id, date>=date_start, date<=date_end)]

If necessary add nomatch=0L to remove non-matching rows in result.

The .(id, x.date) is necessary at the moment until I workout how the default output for a non-equi join should look like.

Upvotes: 3

Frank
Frank

Reputation: 66819

If we transform the OP's data a bit to get

library(data.table)
data = setDT(structure(list(id = 1:10, date = structure(16801:16810, class = c("IDate", 
"Date")), date2 = structure(16801:16810, class = c("IDate", "Date"
))), .Names = c("id", "date", "date2"), row.names = c(NA, -10L
), class = c("data.table", "data.frame"), sorted = c("id", 
"date", "date2")))

query = setDT(structure(list(id = c(1, 4, 7), date_start = 
structure(c(16801L, 
16801L, 16801L), class = c("IDate", "Date")), date_end = structure(c(16804L, 
16802L, 16803L), class = c("IDate", "Date"))), .Names = c("id", 
"date_start", "date_end"), row.names = c(NA, -3L), class = c("data.table", 
"data.frame"), sorted = c("id", 
"date_start", "date_end")))

... then we can use foverlaps like

foverlaps(data, query, nomatch=0)
#    id date_start   date_end       date      date2
# 1:  1 2016-01-01 2016-01-04 2016-01-01 2016-01-01

For this approach, I think one needs to take these steps before merging:

  • have all dates as IDates
  • create an extra date column in the main data
  • set the key on each table

Upvotes: 5

Hack-R
Hack-R

Reputation: 23214

require(data.table)
data = data.table(id = 1:10, date = seq(as.Date("2016-01-01"), by = 1, length = 10))
query = data.table(id = c(1,4,7), date_start = c("2016-01-01", "2016-01-01", 
"2016-01-01"), date_end = c("2016-01-04", "2016-01-02", "2016-01-03"))

First you can just join them:

data.full <- merge(data,query,by="id", all.x=T)

Next, if you want to exclude observations that were not referenced in query and keep the ones that were referenced if they fell in the date range then you can do this:

data.final <- data.full[date >= date_start & date <= date_end,]

data.final
   id       date date_start   date_end
1:  1 2016-01-01 2016-01-01 2016-01-04

or if you want to keep the records that were not referenced in query and keep the ones that were referenced if they fell in the date range:

data.final <- data.full[is.na(date_start) | (date >= date_start & date <= date_end),]
data.final
   id       date date_start   date_end
1:  1 2016-01-01 2016-01-01 2016-01-04
2:  2 2016-01-02         NA         NA
3:  3 2016-01-03         NA         NA
4:  5 2016-01-05         NA         NA
5:  6 2016-01-06         NA         NA
6:  8 2016-01-08         NA         NA
7:  9 2016-01-09         NA         NA
8: 10 2016-01-10         NA         NA

Upvotes: 0

Related Questions