CODEWITHSUNDEEP
nlpmultilingualtext-classificationunsupervised-learningsupervised-learning
yolo25
yolo25

Reputation: 45

Multilingual free-text-items Text Classification for improving a recommender system

To improve the recomender system for Buyer Material Groups, our company is willing to train a model using customer historial spend data. The model should be trained on historical "Short text descriptions" to predict the appropriate BMG. The dataset has more that 500.000 rows and the text descriptions are multilingual (up to 40 characters).

1.Question: can i use supervised learning if i consider the fact that the descriptions are in multiple languages? If Yes, are classic approaches like multinomial naive bayes or SVM suitable?

2.Question: if i want to improve the first model in case it is not performing well, and use unsupervised multilingual emdedding to build a classifier. how can i train this classifier on the numerical labels later?

if you have other ideas or approaches please feel free :). (It is a matter of a simple text classification problem)

Upvotes: 1

Views: 198

Answers (1)

polm23
polm23

Reputation: 15593

Can I use supervised learning if i consider the fact that the descriptions are in multiple languages?

Yes, this is not a problem except it makes your data more sparse. If you actually only have 40 characters (is that not 40 words?) per item, you may not have enough data. Also the main challenge for supervised learning will be whether you have labels for the data.

If Yes, are classic approaches like multinomial naive bayes or SVM suitable?

They will work as well as they always have, though these days building a vector representation is probably a better choice.

If i want to improve the first model in case it is not performing well, and use unsupervised multilingual emdedding to build a classifier. how can i train this classifier on the numerical labels later?

Assuming the numerical labels are labels on the original data, you can add them as tokens like LABEL001 and the model can learn representations of them if you want to make an unsupervised recommender.

Honestly these days I wouldn't start with Naive Bayes or classical models, I'd go straight to word vectors as a first test for clustering. Using fasttext or word2vec is pretty straightforward. The main problem is that if you really only have 40 characters per item, that just might not be enough data to cluster usefully.

Upvotes: 1

Related Questions