CODEWITHSUNDEEP
rtext-mininglevenshtein-distancejaro-winkler
Cam23 19
Cam23 19

Reputation: 19

Text Mining using Jaro-Winkler fuzzy matching in R

Im attempting to do some distance matching in R and am struggling to achieve a usable output.

I have a dataframe terms that contains 5 strings of text, along with a category for each string. I have a second dataframe notes that contains 10 poorly spelt words, along with a NoteID.

I want to be able to compare each of my 5 terms against each of my 10 notes using a distance algorithm to try to grab simple spelling errors. I have tried:

near_match<- subset(notes, jarowinkler(notes$word, terms$word) >0.9) 

   NoteID    Note
5      e5 thought
10     e5   tough

and

jarowinkler(notes$word, terms$word)

[1] 0.8000000 0.7777778 0.8266667 0.8833333 0.9714286 0.8000000 0.8000000 0.8266667 0.8833333 0.9500000

The first instance is almost what I need, it just lacks the word from terms that has caused the match. The second returns 10 scores but I'm not sure if the algorithm checked each of the 5 terms against each of the 10 notes in turn and just returned the closest match (highest score) or not.

How can I alter the above to achieve my desired output if what I want is achievable using jarowinkler() or is there a better option?

I'm relatively new to R so appreciate any help in furthering my understanding how the algorithm generates the scores and what the approach to achieve my desired output would be.

example dataframes below

Thanks

> notes
   NoteID    word
1      a1     hit
2      b2     hot
3      c3   shirt
4      d4    than
5      e5 thought
6      a1     hat
7      b2     get
8      c3   shirt
9      d4    than
10     e5   tough

> terms
  Category   word
1        a    hot
2        b    got
3        a   shot
4        d   that
5        c though

Upvotes: 1

Views: 2169

Answers (1)

Terru_theTerror
Terru_theTerror

Reputation: 5017

Your data.frames:

notes<-data.frame(NoteID=c("a1","b2","c3","d4","e5","a1","b2","c3","d4","e5"),
                  word=c("hit","hot","shirt","than","thought","hat","get","shirt","that","tough"))
terms<-data.frame(Category=c("a","b","c","d","e"),
                  word=c("hot","got","shot","that","though"))

Use stringdistmatrix (package stringdist) with method "jw" (jarowinkler)

library(stringdist)
dist<-stringdistmatrix(notes$word,terms$word,method = "jw")
row.names(dist)<-as.character(notes$word)
colnames(dist)<-as.character(terms$word)

Now you have all distances:

dist
              hot       got       shot       that     though
hit     0.2222222 0.4444444 0.27777778 0.27777778 0.50000000
hot     0.0000000 0.2222222 0.08333333 0.27777778 0.33333333
shirt   0.4888889 1.0000000 0.21666667 0.36666667 0.54444444
than    0.4722222 1.0000000 0.50000000 0.16666667 0.38888889
thought 0.3571429 0.5158730 0.40476190 0.40476190 0.04761905
hat     0.2222222 0.4444444 0.27777778 0.08333333 0.50000000
get     0.4444444 0.2222222 0.47222222 0.47222222 0.50000000
shirt   0.4888889 1.0000000 0.21666667 0.36666667 0.54444444
that    0.2777778 0.4722222 0.33333333 0.00000000 0.38888889
tough   0.4888889 0.4888889 0.51666667 0.51666667 0.05555556

Find the word more close to notes

output<-cbind(notes,word_close=terms[as.numeric(apply(dist, 1, which.min)),"word"],dist_min=apply(dist, 1, min))
output
       NoteID    word word_close   dist_min
    1      a1     hit        hot 0.22222222
    2      b2     hot        hot 0.00000000
    3      c3   shirt       shot 0.21666667
    4      d4    than       that 0.16666667
    5      e5 thought     though 0.04761905
    6      a1     hat       that 0.08333333
    7      b2     get        got 0.22222222
    8      c3   shirt       shot 0.21666667
    9      d4    that       that 0.00000000
    10     e5   tough     though 0.05555556

If you want have just in word_close the words under a certain distance threshold (in this case 0.1), you can do this:

output[output$dist_min>=0.1,c("word_close","dist_min")]<-NA
output
   NoteID    word word_close   dist_min
1      a1     hit       <NA>         NA
2      b2     hot        hot 0.00000000
3      c3   shirt       <NA>         NA
4      d4    than       <NA>         NA
5      e5 thought     though 0.04761905
6      a1     hat       that 0.08333333
7      b2     get       <NA>         NA
8      c3   shirt       <NA>         NA
9      d4    that       that 0.00000000
10     e5   tough     though 0.05555556

Upvotes: 1

Related Questions