CODEWITHSUNDEEP
algorithmmachine-learningnlp
Navaneeth K N
Navaneeth K N

Reputation: 15521

How to extract features from plain text?

I am writing a text parser which should extract features from product descriptions.

Eg:

text = "Canon EOS 7D Mark II Digital SLR Camera with 18-135mm IS STM Lens"
features = extract(text)
print features

Brand: Canon
Model: EOS 7D
....

The way I do this is by training the system with structured data and coming up with an inverted index which can map a term to a feature. This works mostly well.

When the text contains measurements like 50ml, or 2kg, the inverted index will say 2kg -> Size and 50ml -> Size for eg.

The problem here is that, when I get a value which I haven't seen before, like 13ml, it won't be processed. But since the patterns matches to a size, we could tag it as size.

I was thinking to solve this problem by preprocessing the tokens that I get from the text and look for patterns that I know. So when new patterns are identified, that has to be added to the preprocessing.

I was wondering, is this the best way to go about this? Or is there a better way of doing this?

Upvotes: 0

Views: 171

Answers (2)

Ash
Ash

Reputation: 3550

If you have a training data like this:

word label
10ml size-valume
20kg size-weight
etc...

you could train a classifier based on character n-grams and that would detect that ml is size-volume even if it sees a 11-ml or ml11 etc. you should also convert the numbers into a single number (e.g. 0) so that 11-ml is seen as 0-ml before feature extraction.

For that you'll need a preprocessing module and also a large training sample. For feature extraction you can use scikit-learn's character n-grams and also SVM.

Upvotes: 0

flyingmeatball
flyingmeatball

Reputation: 7997

The age-old problem of unseen cases. You could train your scraper to grab any number-like characters preceding certain suffixes (ml, kg, etc) and treat those as size. The problem with this is typos and other poorly formatted texts could enter into your structure data. There is no right answer for how to handle values you haven't seen before - you'll either have to QC them individually, or have rules around them. This is dependent on your dataset.

As far as identifying patterns, you'll either have to manually enter them, or manually classify a lot of records and let the algorithm learn them. Not sure that's very helpful, but a lot of this is very dependent on your data.

Upvotes: 1

Related Questions