Unpacking a list in an R dataframe

Question

I have a dataframe of which one field comprises lists of varying lengths. I would like to extract each element of the list in this field to its own field so that I can gather the results into a long dataframe with each list element per id.

Here is an example dataframe

dat <- structure(list(id = c("509935", "727889", "864607", "1234243", 
        "1020959", "221975"), some_date = c("2/09/1967", "28/04/1976", 
        "22/12/2017", "7/02/2006", "10/03/2019", "21/10/1935"), df_list = list(
            "018084131", c("062197171", "062171593"), c("064601923", 
            "068994009", "069831651"), c("071141584", "073129537"), c("061498574", 
            "065859718", "067251995", "069447806"), "064623976")), class = c("tbl_df", 
        "tbl", "data.frame"), row.names = c(NA, -6L))

I have come with code to achieve what I want the final result to look like, however, I have not done this the DRY way. Here is what I have tried.

res_n is a function as follows:

res_n <- function(field, n) {
    field[n]
}

dat <- dat %>% mutate(res1 = map(df_list, res_n, 1))
dat <- dat %>% mutate(res2 = map(df_list, res_n, 2))
dat <- dat %>% mutate(res3 = map(df_list, res_n, 3))

This returns a data frame with each of the three list elements from df_list in their own columns.

From this I can achieve what I set out to do and produce a final dataframe of results, as follows:

dat_final <- gather(dat, test, labno, -df_list, -some_date, -id) %>% 
    select(-df_list) %>% 
    mutate(labno = as.integer(labno)) %>% 
    filter(!is.na(labno))

To avoid the DRY approach I used I resorted to a for loop to try and eliminate the repetitive code. I'm struggling to get this to work in the way I need to achieve the final result. This is the for loop I tried.

 for (i in 3) {
     dat %>% mutate(paste(res, i, sep = '_') = map(results, res_n, i)) }

Can someone help me to refine the code to elimiate the repeitive lines that produce the result.

Answer 1

Instead of using repeated map, we can make use of unnest_wider

library(dplyr)
library(tidyr)
library(stringr)
out <- dat %>%
         unnest_wider(df_list, names_repair = ~ 
                     str_remove(str_c("res", .x), "[.]+"))
out
# A tibble: 6 x 6
#  id      some_date  res1      res2      res3      res4     
#  <chr>   <chr>      <chr>     <chr>     <chr>     <chr>    
#1 509935  2/09/1967  018084131 <NA>      <NA>      <NA>     
#2 727889  28/04/1976 062197171 062171593 <NA>      <NA>     
#3 864607  22/12/2017 064601923 068994009 069831651 <NA>     
#4 1234243 7/02/2006  071141584 073129537 <NA>      <NA>     
#5 1020959 10/03/2019 061498574 065859718 067251995 069447806
#6 221975  21/10/1935 064623976 <NA>      <NA>      <NA>     

EDIT: Based on @Phil's comments

Now, reshape to 'long' with pivot_longer

out %>% 
    pivot_longer(cols = starts_with('res'), values_drop_na = TRUE) %>%
    mutate(value = as.integer(value))
# A tibble: 13 x 4
#   id      some_date  name     value
#   <chr>   <chr>      <chr>    <int>
# 1 509935  2/09/1967  res1  18084131
# 2 727889  28/04/1976 res1  62197171
# 3 727889  28/04/1976 res2  62171593
# 4 864607  22/12/2017 res1  64601923
# 5 864607  22/12/2017 res2  68994009
# 6 864607  22/12/2017 res3  69831651
# 7 1234243 7/02/2006  res1  71141584
# 8 1234243 7/02/2006  res2  73129537
# 9 1020959 10/03/2019 res1  61498574
#10 1020959 10/03/2019 res2  65859718
#11 1020959 10/03/2019 res3  67251995
#12 1020959 10/03/2019 res4  69447806
#13 221975  21/10/1935 res1  64623976

NOTE: If we check ?unnest, it says the lifecycle as deprecated

nest(.data, ..., .key = deprecated())

unnest(data, cols, ..., keep_empty = FALSE, ptype = NULL, names_sep = NULL, names_repair = "check_unique", .drop = deprecated(), .id = deprecated(), .sep = deprecated(), .preserve = deprecated())

and in ?hoist description is

hoist(), unnest_longer(), and unnest_wider() provide tools for rectangling, collapsing deeply nested lists into regular columns.

---

Also, if the intention is not to get the intermediate wide format, just use unnest_longer

dat %>%
      unnest_longer(df_list)
# A tibble: 13 x 3
#   id      some_date  df_list  
#   <chr>   <chr>      <chr>    
# 1 509935  2/09/1967  018084131
# 2 727889  28/04/1976 062197171
# 3 727889  28/04/1976 062171593
# 4 864607  22/12/2017 064601923
# 5 864607  22/12/2017 068994009
# 6 864607  22/12/2017 069831651
# 7 1234243 7/02/2006  071141584
# 8 1234243 7/02/2006  073129537
# 9 1020959 10/03/2019 061498574
#10 1020959 10/03/2019 065859718
#11 1020959 10/03/2019 067251995
#12 1020959 10/03/2019 069447806
#13 221975  21/10/1935 064623976

---

Or using base R

merge(setNames(stack(setNames(dat$df_list, dat$id))[2:1], 
      c("id", "values")), dat[-3])

Answer 2

Base R solution:

# Split, Apply, Combine Base R: 
# Split the data frame on ids, unlist the dataframe list, replicated the id,
# the number of times as there are elements in the unlisted df list - store
# as a dataframe, left join back to the original data.frame,
# (dropping the df_list vector) using the ID vector, row bind the id data.frames
# back together and store it as a dataframe: 

data.frame(do.call("rbind", lapply(split(df, df$id), function(x){

      unlisted_df_list <- unlist(x$df_list)

      rolled_out_df <- data.frame(id = rep(x$id, length(unlisted_df_list)),

                                 df_list = unlisted_df_list, stringsAsFactors = F)

      x <- merge(x[,names(x) != "df_list"], rolled_out_df, by = "id", all.x = T)

      }

    )

  ),

  row.names = NULL

)

Answer 3

If the final goal is to get data in long format, we can use unnest from tidyr

tidyr::unnest(dat, cols = df_list)

#   id      some_date  df_list  
#   <chr>   <chr>      <chr>    
# 1 509935  2/09/1967  018084131
# 2 727889  28/04/1976 062197171
# 3 727889  28/04/1976 062171593
# 4 864607  22/12/2017 064601923
# 5 864607  22/12/2017 068994009
# 6 864607  22/12/2017 069831651
# 7 1234243 7/02/2006  071141584
# 8 1234243 7/02/2006  073129537
# 9 1020959 10/03/2019 061498574
#10 1020959 10/03/2019 065859718
#11 1020959 10/03/2019 067251995
#12 1020959 10/03/2019 069447806
#13 221975  21/10/1935 064623976