CODEWITHSUNDEEP
rsavesaving-datardata
miamialan
miamialan

Reputation: 73

R needs several hours to save very small objects. Why?

I am running several calculations and ML algorithms in R and store their results in four distinctive tables. For each calculation, I obtain four tables, which I store in a single list. According to R, all of my lists are labelled as "Large List (4 elements, 971.2 kB)" in the upper right quadrant in RStudio where all my objects, functions, etc. are displayed. I have five of these lists and save them for later use with the save() function.

I use the function:

save(list1, list2, list3, list4, list5, file="mypath/mylists.RData")

For some reason, which I do not understand, R takes more than 24 hours to save these four lists with only 971.2 kB each. Maybe, I should add that apparently more than 10GB of my RAM are used by R at the time. However, the lists are as small as I indicated above.

Does anyone have an idea why it takes so long to save the lists to my harddrive and what I could do about it?

Thank you

Upvotes: 1

Views: 658

Answers (2)

miamialan
miamialan

Reputation: 73

Thanks for your answers.

I solved my problem by writing a function which extracts the tables from the objects and saves them as .csv files in a folder. I cleaned the environment and shut down the computer. Afterwards, I restarted the computer, started R and loaded all the .csv files again. I then saved the thus created objects with the familiar save() command. It is probably not the most elegant way, but it worked and was quite quick.

Upvotes: 2

user2554330
user2554330

Reputation: 44957

This is just a guess, because we don't have your data.

Some objects in r contain references to environments. The most common examples are functions and formulas. If you save one of those, r may need to save the whole environment. This can drastically increase the size of what is being saved. If you are short of memory that could take a very long time due to swapping.

Example:

F <- function () {
  X <- rnorm(1000000)
  Y ~ z
}

This function returns a small formula which references the environment holding X, so saving it will take a lot of space.

Upvotes: 2

Related Questions