CODEWITHSUNDEEP
rsplitdataframe
Pascal
Pascal

Reputation: 421

Split up a dataframe by number of rows

I have a dataframe made up of 400'000 rows and about 50 columns. As this dataframe is so large, it is too computationally taxing to work with. I would like to split this dataframe up into smaller ones, after which I will run the functions I would like to run, and then reassemble the dataframe at the end.

There is no grouping variable that I would like to use to split up this dataframe. I would just like to split it up by number of rows. For example, I would like to split this 400'000-row table into 400 1'000-row dataframes. How might I do this?

Upvotes: 41

Views: 49886

Answers (2)

Jurgen Van Impe
Jurgen Van Impe

Reputation: 341

I had a similar question and used this:

library(tidyverse)
n = 100 #number of groups
split <- df %>% group_by(row_number() %/% n) %>% group_map(~ .x)

from left to right:

  • you assign your result to split
  • you start with df as your input dataframe
  • then you group your data by dividing the row_number by n (number of groups) using modular division.
  • then you just pass that group through the group_map function which returns a list.

So in the end your split is a list with in each element a group of your dataset. On the other hand, you could also immediately write your data by replacing the group_map call by e.g. group_walk(~ write_csv(.x, paste0("file_", .y, ".csv"))).

You can find more info on these powerful tools on: Cheat sheet of dplyr explaining group_by and also below for: group_map, group_walk follow up functions

Upvotes: 10

Ben Bolker
Ben Bolker

Reputation: 226871

Make your own grouping variable.

d <- split(my_data_frame,rep(1:400,each=1000))

You should also consider the ddply function from the plyr package, or the group_by() function from dplyr.

edited for brevity, after Hadley's comments.

If you don't know how many rows are in the data frame, or if the data frame might be an unequal length of your desired chunk size, you can do

chunk <- 1000
n <- nrow(my_data_frame)
r  <- rep(1:ceiling(n/chunk),each=chunk)[1:n]
d <- split(my_data_frame,r)

You could also use

r <- ggplot2::cut_width(1:n,chunk,boundary=0)

For future readers, methods based on the dplyr and data.table packages will probably be (much) faster for doing group-wise operations on data frames, e.g. something like

(my_data_frame 
   %>% mutate(index=rep(1:ngrps,each=full_number)[seq(.data)])
   %>% group_by(index)
   %>% [mutate, summarise, do()] ...
)

There are also many answers here

Upvotes: 58

Related Questions