CODEWITHSUNDEEP
routputregressionrows
Vishnu Raghavan
Vishnu Raghavan

Reputation: 83

R multiple regression predict output has more values than contained in the test set

I am trying to train and test a linear regression model in a certain dataset

The following is the header of the training dataset

> head(TaxiTrain)
         id vendor_id     pickup_datetime    dropoff_datetime passenger_count
1 id2875421         2 2016-03-14 17:24:55 2016-03-14 17:32:30               1
2 id2377394         1 2016-06-12 00:43:35 2016-06-12 00:54:38               1
3 id3858529         2 2016-01-19 11:35:24 2016-01-19 12:10:48               1
4 id3504673         2 2016-04-06 19:32:31 2016-04-06 19:39:40               1
5 id2181028         2 2016-03-26 13:30:55 2016-03-26 13:38:10               1
6 id0801584         2 2016-01-30 22:01:40 2016-01-30 22:09:03               6
  pickup_longitude pickup_latitude dropoff_longitude dropoff_latitude
1        -73.98215        40.76794         -73.96463         40.76560
2        -73.98042        40.73856         -73.99948         40.73115
3        -73.97903        40.76394         -74.00533         40.71009
4        -74.01004        40.71997         -74.01227         40.70672
5        -73.97305        40.79321         -73.97292         40.78252
6        -73.98286        40.74220         -73.99208         40.74918
  store_and_fwd_flag trip_duration
1                  N           455
2                  N           663
3                  N          2124
4                  N           429
5                  N           435
6                  N           443

The traning set and contains 1458644 rows

The test set is similar to the training set except for 2 Columns

head(Taxitest)
         id vendor_id     pickup_datetime passenger_count pickup_longitude
1 id3004672         1 2016-06-30 23:59:58               1        -73.98813
2 id3505355         1 2016-06-30 23:59:53               1        -73.96420
3 id1217141         1 2016-06-30 23:59:47               1        -73.99744
4 id2150126         2 2016-06-30 23:59:41               1        -73.95607
5 id1598245         1 2016-06-30 23:59:33               1        -73.97021
6 id0668992         1 2016-06-30 23:59:30               1        -73.99130
  pickup_latitude dropoff_longitude dropoff_latitude store_and_fwd_flag
1        40.73203         -73.99017         40.75668                  N
2        40.67999         -73.95981         40.65540                  N
3        40.73758         -73.98616         40.72952                  N
4        40.77190         -73.98643         40.73047                  N
5        40.76147         -73.96151         40.75589                  N
6        40.74980         -73.98051         40.78655                  N

The test set contains 625134 observations

Now I am facing two problems.I have trained a linear regression model :

lm1 <- lm(trip_duration ~ passenger_count, data = TaxiTrain)

This trains a linear regression model on the training set. When I fit this on the test set I use the following code.

lm2 <- predict(lm1, data = Taxitest)

I get 1458644 observations(Same as the training set). I am supposed to get 625134 predictions

I am not sure where the error is. I Request someone to clarify

Upvotes: 1

Views: 4347

Answers (1)

AntoniosK
AntoniosK

Reputation: 16121

Try to use lm2<-predict(lm1, newdata=Taxitest) instead.

Check how this command works using ?predict.lm. If you don't use newdata= it will predict on the dataset you used to train your model.

As an example see below:

# train and test sets
dt1 = mtcars[1:15,]
dt2 = mtcars[20:23,]

# build the model
lm = lm(disp ~ drat, data = dt1)

# check the differences / similarities
predict(lm, data=dt2)
predict(lm, newdata=dt2)
predict(lm, dt2)

Upvotes: 3

Related Questions