CODEWITHSUNDEEP
nlpdata-miningtext-processingdata-scrubbing
dev.e.loper
dev.e.loper

Reputation: 36034

Open source projects for email scrubbing generating structured data from unstructured source?

Don't know where to start on this one so hopefully you guys can clear up my question. I have project where email will be searched for specific words/patterns and stored in a structured manner. Something that is done with Trip it.

The article states that they developed a DataMapper

The DataMapper is responsible for taking inbound email messages addressed to plans [at] tripit.com and transforming them from the semi-structured format you see in your mail reader into a highly structured XML document.

There is a comment that also states

If you're looking to build this yourself, reading a little bit about Wrappers and Wrapper Induction might be helpful

I Googled and read about wrapper induction but it was just too broad of a definition and didn't help me understand how one would go about solving such problem.

Is there some open source project out there that does similar things?

Upvotes: 0

Views: 828

Answers (2)

Mark Tozzi
Mark Tozzi

Reputation: 10893

Really a very broad question, but I can try to give you some general ideas, which might be enough to get started. Basically, it sounds like you're talking about an elaborate parsing problem - scanning through the text and looking to apply meaning to specific chunks. Depending on what exactly you're looking for, you might get some good mileage out of a few regular expressions to start - things like phone numbers, email addresses, and dates have fairly standard structures that should be matchable. Other data points might benefit from some indicator words - the phrase "departing from" might indicate that what follows is an address. The natural language processing community also has a large tool set available for text processing - check out things like parts of speech taggers and semantic analyzers if they're appropriate to what you're trying to do.

Armed with those techniques, you can follow a basic iterative development process: For each data point in your expected output structure, define some simple rules for how to capture it. Then, run the application over a batch of test data and see which samples didn't capture that datum. Look at the samples and revise your rules to catch those samples. Repeat until the extractor reaches an acceptable level of accuracy.

Depending on the specifics of your problem, there may be machine learning techniques that can automate much of that process for you.

Upvotes: 1

S Kunath
S Kunath

Reputation: 41

There are a couple of different ways and things you can do to accomplish this.

The first part, which involves getting access to the email content I'll not answer here. Basically, I'll assume that you have access to the text of emails, and if you don't there are some libraries that allow you to connect java to an email box like camel (http://camel.apache.org/mail.html).

So now you've got the email so then what?

A handy thing that could help is that lingpipe (http://alias-i.com/lingpipe/) has an entity recognizer that you can populate with your own terms. Specifically, look at some of their extraction tutorials and their dictionary extractor (http://alias-i.com/lingpipe/demos/tutorial/ne/read-me.html) So inside of the lingpipe dictionary extractor (http://alias-i.com/lingpipe/docs/api/com/aliasi/dict/ExactDictionaryChunker.html) you'd simply import the terms you're interested in and use that to associate labels with an email.

You might also find the following question helpful: Dictionary-Based Named Entity Recognition with zero edit distance: LingPipe, Lucene or what?

Upvotes: 1

Related Questions