CODEWITHSUNDEEP
rperformancefor-loopparallel-processing
Sharif Amlani
Sharif Amlani

Reputation: 1278

Is there a way to speed up a for loop?

I am working with a for loop. The goal of the for loop is to simply test a condition and code the data appropriately. The loop has to iterate over 503,288 unique values and includes three if statements. Is there a way to speed up the for loop?

The code is as follows:

count<- 0
for(i in unique(Data$ID)){ #503288

  #Subset Relevant Data
  Loop_Before<- subset(Primary_Before, ID == i); Loop_After <- subset(Primary_After, ID == i)

  if(nrow(Loop_Before) >= 1 & nrow(Loop_After) >= 1){
    Data$Status[Data$ID == i] <- "Both Elections"

  }

  if(nrow(Loop_Before) >= 1 & nrow(Loop_After) == 0){
    Data$Status[Data$ID == i] <- "Only Primary Election"

  }

  if(nrow(Loop_Before) >= 0 & nrow(Loop_After) == 1){
    Data$Status[Data$ID == i] <- "Only General Election"

  }

  message(count<- count +1)

}



table(Data$Status)

Thank you for your help!

Upvotes: 0

Views: 123

Answers (1)

Georgery
Georgery

Reputation: 8117

Avoid the for-loop entirely. I don't know your dataset but the following should be 10 or even 100 times faster:

library(tidyverse) # load some packages that will help

# let's create some sample data
Data <- data.frame(ID = c(1,1,1,1,2,2,2,3,3))
Primary_before <- data.frame(ID = c(0,1,2,2,3,3,3))
Primary_after <- data.frame(ID = c(1,3))

# now for every ID we count the nr of rows in both dataframes
summarised_before <- Primary_before %>%
    group_by(ID) %>%
    summarise(nrRows = n())

     ID nrRows
  <dbl>  <int>
1     0      1
2     1      1
3     2      2
4     3      3

summarised_after <- Primary_after %>%
    group_by(ID) %>%
    summarise(nrRows = n())

     ID nrRows
  <dbl>  <int>
1     1      1
2     3      1

# now we join them together
summarised_both <- summarised_after %>%
    full_join(summarised_before, by = "ID", suffix = c("_after", "_before"))

# and now we do the final calculation
summarised_both %>%
    mutate(nrRows_after = replace_na(nrRows_after, 0)) %>%
    mutate(Status = case_when(nrRows_before >= 1 & nrRows_after >= 1    ~ "Both elections"
                              , nrRows_before >= 1 & nrRows_after == 0  ~ "Only primary election"
                              , nrRows_before >= 0 & nrRows_after == 1  ~ "Only general election")) %>%
    filter(ID %in% Data$ID)

I saved the intermediate results, but you could also do it in one go, like this:

Primary_before %>%
    group_by(ID) %>%
    summarise(nrRows = n()) %>%
    full_join(Primary_after %>%
                  group_by(ID) %>%
                  summarise(nrRows = n())
              , by = "ID"
              , suffix = c("_after", "_before")) %>%
    mutate(nrRows_after = replace_na(nrRows_after, 0)) %>%
    mutate(Status = case_when(nrRows_before >= 1 & nrRows_after >= 1    ~ "Both elections"
                              , nrRows_before >= 1 & nrRows_after == 0  ~ "Only primary election"
                              , nrRows_before >= 0 & nrRows_after == 1  ~ "Only general election")) %>%
    filter(ID %in% Data$ID)

Upvotes: 4

Related Questions