CODEWITHSUNDEEP
scalaloopscsvapache-sparknested
Eyedia Tech
Eyedia Tech

Reputation: 145

Scala Nested Iteration within RDD

I have to iterate through all columns to find similarity of 1 column value. For example:

ID,FN,LN,Phone
-----------
1,James,Butt,872-232-1212
2,Josephine,Darakjy, 872-232-1213
3,Art,Venere,872-232-1214
4,Lenna,Paprocki,872-232-1215
5,Donette, Foller,872-232-1216
6,Jmes,Butt,666-232-1212
7,Donette, Foller,888-232-1216
8,Josphne,Darkjy, 555-232-1213

Inside the loop, I will take FN, which is 'James' and see if I have similar name in the complete data set using some kind string distances (e.g Levenshtein) and in this case I have match with ID#6: 'Jmes', I will create a bucket by adding a new GUID column this:

ID,FN,LN,Phone,GrupId
----------------------
1,James,Butt,872-232-1212,G1
2,Josephine,Darakjy, 872-232-1213,G2
3,Art,Venere,872-232-1214,G3
4,Lenna,Paprocki,872-232-1215,G4
5,Donette, Foller,872-232-1216,G5
6,Jmes,Butt,666-232-1212,G1
7,Donette, Foller,888-232-1216,G5
8,Josphne,Darkjy, 555-232-1213,G2

I have to do same operation on multiple columns, like LN, Phone as well. Imagine if I have 1 million records.

Any thoughts, suggestions or links are appreciated. Thank you!

Upvotes: 1

Views: 129

Answers (1)

bobah
bobah

Reputation: 18864

I would definitely not try anything pairwise and would rather think towards coding a per-field Levenshtein-y index and accumulate results on the fly. I’d probably start from a suffix tree -ish one. Will try to sketch a prototype as soon as I get to the laptop...

Update: after some reading I am leaning towards Affinity Clustering1 combined with pairwise (yes I know) Levenshtein cached on a Trie2. Code in progress...

Upvotes: 1

Related Questions