CODEWITHSUNDEEP
rtidyversepurrr
Matt
Matt

Reputation: 11

R loop a spred over list

A Hey I have a massive dataset I'm trying to spread in R and I keep running out of memory so trying to break it up and to write an efficent piece of code to loop over the list example structure below. I know#this will be dead easy to someone who understands these functions and appreciate any suggestions.

rs <- split(r,1:3)

Then spread the lists as below, but do so efficiently as there may be a large number of lists

rs$'1' <- rs$'1' %>% spread(movieId, resids)
rs$'2' <- rs$'2' %>% spread(movieId, resids)
rs$'3' <- rs$'3' %>% spread(movieId, resids)

Upvotes: 1

Views: 80

Answers (1)

Georgery
Georgery

Reputation: 8117

Considering the comments above, I would say:

  1. Your dataset is not too big. 144 MB is totally handable unless your computer is from 1997.

  2. spread() tries to create a dataframe with 10.000 rows. I think this is the reason, you're running out of memory.

The resulting point is: what do you eventually want to do with your data. It's likely that you actually not want to do spread(), but something else really.

Upvotes: 1

Related Questions