CODEWITHSUNDEEP
r
Will C.
Will C.

Reputation: 43

Applying a set of operations across several data frames in r

I've been learning R for my project and have been unable to google a solution to my current problem.

I have ~ 100 csv files and need to perform an exact set of operations across them. I've read them in as separate objects (which I assume is probably improper r style) but I've been unable to write a function that can loop through. Each csv is a dataframe that contain information, including a column with dates in decimal year form. I need to create 2 new columns containing year and day of year. I've figured out how to do it manually I would like to find a way to automate the process. Here's what I've been doing:

#setup
library(lubridate)  #Used to check for leap years
df.00 <- data.frame( site = seq(1:10), date = runif(10,1980,2000 ))


#what I need done

df.00$doy <- NA  # make an empty column which I'm going to place the day of the year
df.00$year <- floor(df.00$date) # grabs the year from the date column
df.00$dday <- df.00$date - df.00$year # get the year fraction. intermediate step.

# multiply the fraction year by 365 or 366 if it's a leap year to give me the day of the year
df.00$doy[which(leap_year(df.00$year))] <- round(df.00$dday[which(leap_year(df.00$year))] * 366)
df.00$doy[which(!leap_year(df.00$year))] <- round(df.00$dday[which(!leap_year(df.00$year))] * 365)

The above, while inelegant, does what I would like it to. However, I need to do this to the other data frames, df.01 - df.99. So far I've been unable to place it in a function or for loop. If I place it into a function:

funtest <- function(x) {
    x$doy <- NA
}

funtest(df.00) does nothing. Which is what I would expect from my understanding of how functions work in r but if I wrap it up in a for loop:

for(i in c(df.00)) { 
    i$doy <- NA }

I get "In i$doy <- NA : Coercing LHS to a list" several times which tells me that the loop isn't treat the dataframe as a single unit but perhaps looking at each column in the frame.
I would really appreciate some insight on what I should be doing. I feel that I could have solved this easily using bash and awk but I would like to be less incompetent using r

Upvotes: 4

Views: 2195

Answers (1)

Ricardo Saporta
Ricardo Saporta

Reputation: 55340

the most efficient and direct way is to use a list.

  1. Put all of your CSV's into one folder
  2. grab a list of the files in that folder
    eg: files <- dir('path/to/folder', full.names=TRUE)
  3. iterativly read in all those files into a list of data.frames
    eg: df.list <- lapply(files, read.csv, <additional args>)
  4. apply your function iteratively over each data.frame eg: lapply(df.list, myFunc, <additional args>)

Since your df's are already loaded, and they have nice convenient names, you can grab them easily using the following: 

nms     <- c(paste0("df.0", 0:9), paste0("df.", 10:99))
df.list <- lapply(nms, get)

Then take everything you have in the #what I need done portion and put inside a function, eg: 

myFunc <- function(DF) {

   # what you want done to a single DF
return(DF)
}

And then lapply accordingly

df.list <- lapply(df.list, myFunc)

On a separate notes, regarding functions:

The reason your funTest "does nothing" is that it you are not having it return anything. That is to say, it is doing something, but when it finishes doing that, then it does "nothing".

You need to include a return(.) statement in the function. Alternatively, the output of last line of the function, if not assigned to an object, will be used as the return value -- but this last sentence is only loosely true and hence one needs to be cautious. The cleanest option (in my opinion) is to use return(.)

regarding the for loop over the data.frame

As you observed, using for (i in someDataFrame) {...} iterates over the columns of the data.frame.

You can iterate over the rows using apply: 

apply(myDF, MARGIN=1, function(x) { x$doy <- ...; return(x) } ) # dont forget to return

Upvotes: 5

Related Questions