R - grepl over 7 million observations - how to boost efficiency?

Question

I've run into a bit of a dead end with some R code that I've written, and I thought maybe you'd know how to make this whole thing feasible, in the sense that efficiency can be improved.

So, what I'm trying to do is the following:

I've got a tweet dataset with ~7 million observations. Currently, I'm not interested in the text of the tweets or any of the other metadata, but only in the "Location" field, so I've extracted that data into a new data.frame, which contains the location variable (string) and a new, currently empty, "isRelevant" variable (logical). Furthermore, I've got a vector containing text information formatted as follows: "Placename(1)|Placename(2)[...]|Placename(i)" . What I'm trying to do is to grepl every line of the locations variable to see if there is a match with the Placenames vector, and if so, return a "TRUE" in the isRelevant variable and return a "FALSE" if not.

To do this, I wrote some R code, which basically boils down to this line:

locations.df$isRelevant <- sapply(locations.df$locations, function(s) grepl(grep_places, s, ignore.case = TRUE))

whereby grep_places is the list of possible matching terms separated by "|" characters, to let R know that it can match any of the elements in the vector. I am running this on a remote high-capacity computer, which provides over 2 TB of RAM using RStudio (R 3.2.0), and I'm running it with 'pbsapply' which provides me with a progress bar. As it turns out, this is taking ridiculously long. It's done about 45% to date (I started it more than a week ago) and it's saying it's still going to need over 270 hours to complete it. That's obviously not really a workable situation, as I'm going to have to run similar code in the future, using way larger datasets. Have you got any idea how I might get this job done in a more acceptable timeframe, perhaps like one day or something like that (keeping in mind the super-strong computer).

EDIT

Here's some semi-simulated data to indicate what I'm working with approximately looks like:

print(grep_places)
> grep_places
"Acworth NH|Albany NH|Alexandria NH|Allenstown NH|Alstead NH|Alton NH|Amherst NH|Andover NH|Antrim NH|Ashland NH|Atkinson NH|Auburn NH|Barnstead NH|Barrington NH|Bartlett NH|Bath NH|Bedford NH|Belmont NH|Bennington NH|Benton NH|Berlin NH|Bethlehem NH|Boscawen NH|Bow NH|Bradford NH|Brentwood NH|Bridgewater NH|Bristol NH|Brookfield NH|Brookline NH|Campton NH|Canaan NH|Candia NH|Canterbury NH|Carroll NH|CenterHarbor NH|Charlestown NH|Chatham NH|Chester NH|Chesterfield NH|Chichester NH|Claremont NH|Clarksville NH|Colebrook NH|Columbia NH|Concord NH|Conway NH|Cornish NH|Croydon NH|Dalton NH|Danbury NH|Danville NH|Deerfield NH|Deering NH|Derry NH|Dorchester NH|Dover NH|Dublin NH|Dummer NH|Dunbarton NH|Durham NH|EastKingston NH|Easton NH|Eaton NH|Effingham NH|Ellsworth NH|Enfield NH|Epping NH|Epsom NH|Errol NH|Exeter NH|Farmington NH|Fitzwilliam NH|Francestown NH|Franconia NH|Franklin NH|Freedom NH|Fremont NH|Gilford NH|Gilmanton NH|Gilsum NH|Goffstown NH|Gorham NH|Goshen NH|Grafton NH|Grantham NH|Greenfield NH|Greenland NH|Greenville NH|Groton NH|Hampstead NH|Hampton NH|HamptonFalls NH|Hancock NH|Hanover NH|Harrisville NH|Hart'sLocation NH|Haverhill NH|Hebron NH|Henniker NH|Hill NH|Hillsborough NH|Hinsdale NH|Holderness NH|Hollis NH|Hooksett NH|Hopkinton NH|Hudson NH|Jackson NH|Jaffrey NH|Jefferson NH|Keene NH|Kensington NH|Kingston NH|Laconia NH|Lancaster NH|Landaff NH|Langdon NH|Lebanon NH|Lee NH|Lempster NH|Lincoln NH|Lisbon NH|Litchfield NH|Littleton NH|Londonderry NH|Loudon NH|Lyman NH|Lyme NH|Lyndeborough NH|Madbury NH|Madison NH|Manchester NH|Marlborough NH|Marlow NH|Mason NH|Meredith NH|Merrimack NH|Middleton NH|Milan NH|Milford NH|Milton NH|Monroe NH|MontVernon NH|Moultonborough NH|Nashua NH|Nelson NH|NewBoston NH|NewCastle NH|NewDurham NH|NewHampton NH|NewIpswich NH|NewLondon NH|Newbury NH|Newfields NH|Newington NH|Newmarket NH|Newport NH|Newton NH|NorthHampton NH|Northfield NH|Northumberland NH|Northwood NH|Nottingham NH|Orange NH|Orford NH|Ossipee NH|Pelham NH|Pembroke NH|Peterborough NH|Piermont NH|Pittsburg NH|Pittsfield NH|Plainfield NH|Plaistow NH|Plymouth NH|Portsmouth NH|Randolph NH|Raymond NH|Richmond NH|Rindge NH|Rochester NH|Rollinsford NH|Roxbury NH|Rumney NH|Rye NH|Salem NH|Salisbury NH|Sanbornton NH|Sandown NH|Sandwich NH|Seabrook NH|Sharon NH|Shelburne NH"


head(location.df, n=20)
>                      location isRelevant
1                      London         NA
2      Orleans village VT USA         NA
3                   The World         NA
4               D M V Towson          NA
5   Playa del Sol Solidaridad         NA
6  Beautiful Downtown Burbank         NA
7                        <NA>         NA
8                          US         NA
9             Gaithersburg Md         NA
10                       <NA>         NA
11                California          NA
12                       Indy         NA
13                    Florida         NA
14              exsnaveen com         NA
15                 Houston TX         NA
16                   Tweaking         NA
17                Phoenix AZ          NA
18              Malibu Ca USA         NA
19           Hermosa Beach CA         NA
20             California USA         NA

Thanks in advance everyone, I'd seriously appreciate any help with this.

Answer 1

grepl is a vectorized function, there should be no need to apply a loop to it. Have you tried:

#dput(location.df)    
location.df<-structure(list(location = structure(c(12L, 14L, 17L, 5L, 16L, 
          2L, 1L, 19L, 8L, 1L, 3L, 11L, 7L, 6L, 10L, 18L, 15L, 13L, 9L, 
         4L), .Label = c("<NA>", "Beautiful Downtown Burbank", "California", 
          "California USA", "D M V Towson", "exsnaveen com", "Florida", 
          "Gaithersburg Md", "Hermosa Beach CA", "Houston TX", "Indy", 
          "London", "Malibu Ca USA", "Orleans village VT USA", "Phoenix AZ", 
          "Playa del Sol Solidaridad", "The World", "Tweaking", "US"), class = "factor"), 
           isRelevant = c(NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
           NA, NA, NA, NA, NA, NA, NA, NA, NA, NA)), .Names = c("location", 
          "isRelevant"), row.names = c(NA, -20L), class = "data.frame")

#grep_places with places in the test data
grep_places<-"Gaithersburg Md|Phoenix AZ"

location.df$isRelevant[grepl(grep_places, location.df$location, ignore.case = TRUE)]<-TRUE

or for a slightly faster implementation,as per David Arenburg's comment:

location.df$isRelevant <- grepl(grep_places, location.df$location, ignore.case = TRUE)