CODEWITHSUNDEEP
nlp
Michael Bishop
Michael Bishop

Reputation: 1935

nlp: alternate spelling identification

Help by editing my question title and tags is greatly appreciated!

Sometimes one participant in my corpus of "conversations" will refer to another participant using a nickname, usually an abbreviation or misspelling, but hereafter I'll just say "nicknames". Let's say I'm willing to manually tell my software whether or not I think various possible nicknames are in fact nicknames, but I want software to come up with a list of possible matches between the handle's that identify people, and the potential nicknames. How would I go about doing that?

Background on me and then my corpus: I have no experience doing natural language processing but I'm a competent data analyst with R. My data is produced by 70 teams, each forecasting the likelihood of 100 distinct events occurring some time in the future. The result that I have 70 x 100 = 7000 text files, containing the stream of forecasts participants make and the comments they include with their forecasts. I'll paste a very short snip of one of these text files below, this one had to do with whether the Malian government would enter talks with the MNLA:

02/12/2013 20:10: past_returns answered Yes: (50%)

I hadn't done a lot of research when I put in my previous placeholder... I'm bumping up a lot due to DougL's forecast

02/12/2013 19:31: DougL answered Yes: (60%)

Weak President Traore wants talks if MNLA drops territorial claims. Mali's military may not want talks. France wants talks. MNLA sugggests it just needs autonomy. But in 7 weeks?

02/12/2013 10:59: past_returns answered No: (75%)

placeholder forecast... http://www.irinnews.org/Report/97456/What-s-the-way-forward-for-Mali

My initial thoughts: Obviously I can start by providing the names I'm looking to match things up with... in the above example they would be past_returns and DougL (though there is no use of nicknames in the above). I wouldn't think it'd be that hard to get a computer to guess at minor misspellings (though I wouldn't personally know where to start). I can imagine that other tricks could be used, like assuming that a string is more likely to be a nickname if it is used much much more by one team, than by other teams. A nickname is more likely to refer to someone who spoke recently than someone who spoke long ago, or not at all on regarding this question. And they should be used in sentences in a manner similar to the way the full name/screenname is typically used in the corpus. But I'm interested to hear about simple approaches, as well as ones that try to consider more sophisticated techniques.

Upvotes: 2

Views: 595

Answers (1)

Josh
Josh

Reputation: 1553

This could get about as complicated as you want to make it. From the semi-linguistic side of things, research topics would include Levenshtein Distance (for detecting minor misspellings of known names/nicknames) and Named Entity Recognition (for the task of detecting names/nicknames in the first place). Actually, NER's worth reading about, but existing systems might not help you much in your domain of forum handles and nicknames.

The first rough idea that comes to mind is that you could run a tokenized version of your corpus against an English dictionary (perhaps a dataset compiled from Wiktionary or something like WordNet) to find words that are candidates for names, then filter those through some heuristics (do they start with the same letters as known full names? Do they have a low Levenshtein distance from known names? Are they used more than once?).

You could also try some clustering or supervised ML algorithms against the non-word tokens. That might reveal some non-"word" tokens that often occur in the same threads as a given username; again, heuristics could help rule out some false positives.

Good luck; sounds like a fun problem - hope I mentioned at least one thing you hadn't already thought of.

Upvotes: 3

Related Questions