CODEWITHSUNDEEP
rndjson
sandrine huart
sandrine huart

Reputation: 21

how-to-import-multiple-ndjson-files-into-r-as-a-dataframe?

I need to open 100 ndjson large files (with same columns) , I have prepared a script to apply to each file but I would not like to repeat this 100 times !

With ndjson::stream_in , I can only open 1 ndjson file into R as a data frame

I tried the process to open multiple csv files and consolidate them into 1 dafatframe only, but it does not work with ndjson files :(

library(data.table)
library(purrr)

map_df_fread <- function(path, pattern = "*.ndjson") {
    list.files(path, pattern, full.names = TRUE) %>% 
    map_df(~fread(., stringsAsFactors = FALSE))
}


myfiles <-
    list.files(path = "C:/Users/sandrine/Documents/Projet/CAD/A/",
               pattern = "*.ndjson", 
               full.names = T) %>% 
    map_df_fread(~fread(., stringsAsFactors = FALSE))

I tried to find also a package to convert ndjson files into csv ...but did not find any.

Any idea?

Upvotes: 2

Views: 208

Answers (1)

user63230
user63230

Reputation: 4698

Using your own approach that you mentioned first, does this work?

library(tidyverse)
library(ndjson)
final_df <- 
  list.files(path = "C:/Users/sandrine/Documents/Projet/CAD/A/",
           pattern = "*.ndjson", 
           full.names = T) %>% 
  map_dfr(~stream_in(.))

Upvotes: 1

Related Questions