What are the restrictions for the column classes in data.table?

Question

UPDATE This problem is not relevant anymore for data.table versions 1.8.0 and higher. From the NEWS file:

character columns are now allowed in keys and are preferred to factor. data.table() and setkey() no longer coerce character to factor. Factors are still supported. Implements FR#1493, FR#1224 and (partially) FR#951.

Original question

I try to join two data.tables. However, the success of the join is dependent on the classes of the columns I use to match the data.tables. More precisely, it seems that the columns should not have the class "character". I don't quite understand the reason, but I'm sure I'm missing something obvious here. So help is really appreciated.

Here is an example:

#Objective: Select all rows from DT for which Region=="US", Year >= 5 & Year<=8, Cat="A"                 
library(data.table)
#Set-up data.table DT
DT <- data.table(Year=1:20, value=rnorm(20), Region=c(rep("US", 10), rep("EU", 10)),     Cat=c(rep("A", 7), rep("B", 7), rep("C", 6)))
setkey(DT, Region, Cat, Year)
#Set-up data.table int_DT to join with DT
years   <- 5:8
df      <- data.frame(Region=c("US", "EU"), Categ=c("A", "B"))
int_DT <- J(cbind(df[1, ], years))
#Join them: Works like a charm!
DT[int_DT]

#Let's assume that for any reason the columns in df are of class "character"
df$Region <- as.character(df$Region)
df$Categ  <- as.character(df$Categ)
#Rebuild int_DT
int_DT    <- J(cbind(df[1, ], years))
DT[int_DT]    
#Error in `[.data.table`(DT, int_DT) : 
#  unsorted column Region of i is not internally type integer.

#OK, maybe the problem is that the column classes in DT are factors, so change those:
DT[, Cat:=as.character(Cat)]
DT[, Region:=as.character(Region)]

DT[int_DT]
#Error in `[.data.table`(DT, int_DT) : 
#  When i is a data.table, x must be sorted to avoid a vector scan of x per row of i

Still doesn't work. Why? What is the restriction? What do I miss? Additionally information: I'm using data.table 1.6.6 and R version 2.13.2 (2011-09-30) on Platform: x86_64-pc-linux-gnu (64-bit).

Answer 1

You don't need a join operation to get your desired results. You said: 'Objective: Select all rows from DT for which Region=="US", Year >= 5 & Year<=8, Cat="A"'

DT[Region=="US" & Year>=5 & Year <= 8 & Categ=="A"]
     Year       value Region Categ
[1,]    5 -0.18631697     US     A
[2,]    6  1.40059083     US     A
[3,]    7  0.01848557     US     A

---

But to answer your question about column classes. I managed to get this code to work, which essentially mirrors your code above:

> setkey(DT, Region, Categ, Year)
> df      <- data.frame(Region=c("US", "EU"), Categ=c("A", "B"))
> dt2 <- data.table(data.frame(df[1, ], Year=5:8))
Warning message:
In data.frame(df[1, ], Year = 5:8) :
  row names were found from a short variable and have been discarded
> dt1[dt2]
     Region Categ Year      value
[1,]     US     A    5 -0.5565422
[2,]     US     A    6 -0.1805841
[3,]     US     A    7  1.4474403
[4,]     US     A    8         NA

---

The same, with column classes of character:

df$Region <- as.character(df$Region)
df$Categ  <- as.character(df$Categ)
#Rebuild int_DT
dt2    <- J(cbind(df[1, ], Year=5:8))

Warning message:
In data.frame(..., check.names = FALSE) :
  row names were found from a short variable and have been discarded

setkey(dt2, Region)
dt1[dt2]
   Region Year       value Categ Categ.1 Year.1
       US    1  1.20152558     A       A      5
       US    2  1.89391079     A       A      5
       US    3 -1.76022634     A       A      5
       US    4  0.92454680     A       A      5
       US    5 -0.55654217     A       A      5
       ...
       snip 
       ...
       US    9  0.67936243     B       A      8
       US   10 -0.09355764     B       A      8