CODEWITHSUNDEEP
regexnlp
rich
rich

Reputation: 605

natural language processing fix for combined words

I have some text that was generate by another system. It combined some words together in what I assume was some sort of wordwrap by-product. So something simple like 'the dog' is combine into 'thedog'.

I checked the ascii and unicode string to see is there wasn't some unseen character in there, but there wasn't. A confounding problem is that this is medical text and a corpus to check against aren't that available. So, real example is '...test to rule out SARS versus pneumonia' ends up as '... versuspneumonia.'

Anyone have a suggestion for finding and separating these?

Upvotes: 7

Views: 1105

Answers (3)

rich
rich

Reputation: 605

Here is what I did. I combined a couple of ideas and using a general bootstrapping methodology came up with a pretty good solution. I used Python for all of this.

  1. took a sample of reports, tokenized all the words and created a frequency table.
  2. For words with a frequency of 3 or under (frequency of 4 or more was deemed common enough to be correct), I spell checked them using PyEnchant package (enchant library)
  3. built a medical dictionary from the 'misspelled' words, in step 2, that were clinical.
  4. for all the reports, created a frequency table
  5. for words with a frequency under 4, I spell checked each using PyEnchant and my medical dictionary
  6. Took each misspelled word and split them in all possible ways. The splits were tested for the creation of 2 correctly spelled words. kept any successful split
  7. For each potential solutions the highest weighted solution was used.

Upvotes: 0

Finbar Crago
Finbar Crago

Reputation: 444

This may be of interest to you http://www.perlmonks.org/?node_id=336331

You can probably use the medical nature of the text to your advantage by using two dictionaries, one containing only medical terminology and one of general English.

If you can isolate out medical words then run the rest of the string against the general dictionary you should get some decent results.

Upvotes: 2

zebediah49
zebediah49

Reputation: 7611

This is a rather tricky problem.

I would probably say a combination method is your best bet.

  1. Search for "misspelled words"
  2. For each one of these, check to see if there is some combination of dictionary words which can make it. You can assume that a word is only made up of two words, because of step 4 2.1. If you get a match, confirm with the human.
  3. If there is no match, ask the human to say "this is a real word you don't have", or "this is the correction"

It'd pretty much be an advanced form of spellcheck. You could automate it more, but I'd not risk it on something that important.

Alternatively, you can look for patterns with when the breaks happen. Thus if, for example, every nth character that should be a space isn't, you can fix that.

Upvotes: 1

Related Questions