CODEWITHSUNDEEP
rdataframe
Fenil
Fenil

Reputation: 396

Model.Matrix returning infinite values

I have a data frame with no NA's(removed by na.omit()) with number of rows = 1,450,683 , while converting it into model.matrix to feed into glmnet the final matrix has some values that are Infinite.

 str(train_again)
 Classes 'tbl_df', 'tbl' and 'data.frame':  1450683 obs. of  24  variables:
$ vendor_id          : Factor w/ 2 levels "1","2": 2 1 2 2 2 2 1 2 1 2 ...
$ passenger_count    : int  1 1 1 1 1 6 4 1 1 1 ...
$ pickup_longitude   : num  -74 -74 -74 -74 -74 ...
$ pickup_latitude    : num  40.8 40.7 40.8 40.7 40.8 ...
$ dropoff_longitude  : num  -74 -74 -74 -74 -74 ...
$ dropoff_latitude   : num  40.8 40.7 40.7 40.7 40.8 ...
$ store_and_fwd_flag : Factor w/ 2 levels "N","Y": 1 1 1 1 1 1 1 1 1 1 ...
$ trip_duration      : int  455 663 2124 429 435 443 341 1551 255 1274 ...
$ month              : Factor w/ 6 levels "1","2","3","4",..: 3 6 1 4 3 1 6 5 5 5 ...
$ wday               : Factor w/ 7 levels "Fri","Mon","Sat",..: 2 4 6 7 3 3 1 3 1 6 ...
$ hour               : int  17 0 11 19 13 22 22 7 23 22 ...
$ work               : Factor w/ 2 levels "FALSE","TRUE": 2 1 2 1 1 1 1 1 1 1 ...
$ jfk_trip           : Factor w/ 2 levels "FALSE","TRUE": 1 1 1 1 1 1 1 1 1 1 ...
$ lg_trip            : Factor w/ 2 levels "FALSE","TRUE": 1 1 1 1 1 1 1 1 1 1 ...
$ average.temperature: num  45.5 72.5 22 39 46.5 33.5 70.5 60 80 56.5 ...
$ rain               : num  25 2 2 2 2 2 2 6 2 2 ...
$ s_fall             : num  2 2 2 2 2 2 2 2 2 2 ...
$ s_depth            : num  1 1 0.01 1 1 8 1 1 1 1 ...
$ total_distance     : num  2009 2513 11061 1779 1615 ...
$ number_of_steps    : int  5 6 16 4 5 5 5 17 2 6 ...
$ fastest_speed      : num  43.9 27.3 51.9 27.2 41.5 ...
$ left_turns         : int  1 2 5 2 2 1 1 4 0 2 ...
$ right_turns        : int  1 2 7 1 2 3 3 9 1 2 ...
$ turns              : int  1 2 9 1 3 3 2 6 0 3 ...


 x = model.matrix(trip_duration~.,train_again) #here train_again is a data frame with no NA's
 y = train_again$trip_duration
 sum(is.infinite(x)) #gives output as 537

What could be the possible reason? Must there be something wrong with my original dataset?

Upvotes: 0

Views: 96

Answers (1)

Ben Bolker
Ben Bolker

Reputation: 226936

Given your comments that one of your variables has an Inf (not NA) value, I would recommend something like:

(Using tidyverse since you already show you have a tibble.) Drop response variable:

predvars <- dplyr::select(train_again,-trip_duration)

Find all-finite rows (no NA, NaN, or Inf):

all_finite <- apply(is.finite(predvars),1,all)

You can use a one-sided formula in model.matrix():

x <- model.matrix(~.,predvars[all_finite,])

Upvotes: 1

Related Questions