tfidf first time, using it on a Pandas series that has a list per entry

Question

Data looks like this :

data_clean2.head(3)

text    target
0   [deed, reason, earthquak, may, allah, forgiv, u]    1
1   [forest, fire, near, la, rong, sask, canada]    1
2   [resid, ask, shelter, place, notifi, offic, evacu, shelter, place, order, expect]   1

I got this by stemming and lemmatizing the sentence and tokenizing before that. ( Hope that is right).

Now I want to use:

vectorizer = TfidfVectorizer()
vectors = vectorizer.fit_transform(data_clean2['text'])

It gives me the following error :

---

AttributeError                            Traceback (most recent call last)
<ipython-input-140-6f68d1115c5f> in <module>
      1 vectorizer = TfidfVectorizer()
----> 2 vectors = vectorizer.fit_transform(data_clean2['text'])

~\Anaconda3\lib\site-packages\sklearn\feature_extraction\text.py in fit_transform(self, raw_documents, y)
   1650         """
   1651         self._check_params()
-> 1652         X = super().fit_transform(raw_documents)
   1653         self._tfidf.fit(X)
   1654         # X is already a transformed view of raw_documents so

~\Anaconda3\lib\site-packages\sklearn\feature_extraction\text.py in fit_transform(self, raw_documents, y)
   1056 
   1057         vocabulary, X = self._count_vocab(raw_documents,
-> 1058                                           self.fixed_vocabulary_)
   1059 
   1060         if self.binary:

~\Anaconda3\lib\site-packages\sklearn\feature_extraction\text.py in _count_vocab(self, raw_documents, fixed_vocab)
    968         for doc in raw_documents:
    969             feature_counter = {}
--> 970             for feature in analyze(doc):
    971                 try:
    972                     feature_idx = vocabulary[feature]

~\Anaconda3\lib\site-packages\sklearn\feature_extraction\text.py in <lambda>(doc)
    350                                                tokenize)
    351             return lambda doc: self._word_ngrams(
--> 352                 tokenize(preprocess(self.decode(doc))), stop_words)
    353 
    354         else:

~\Anaconda3\lib\site-packages\sklearn\feature_extraction\text.py in <lambda>(x)
    254 
    255         if self.lowercase:
--> 256             return lambda x: strip_accents(x.lower())
    257         else:
    258             return strip_accents

AttributeError: 'list' object has no attribute 'lower'

I know that I somehow cannot use it on the list, so what is my play here, trying to return the list into a string again?

Answer 1

Yes, first convert to string using:

data_clean2['text'] = data_clean2['text'].apply(', '.join)

Then use:

vectorizer = TfidfVectorizer()
vectors = vectorizer.fit_transform(data_clean2['text'])
v = pd.DataFrame(vectors.toarray(), columns = vectorizer.get_feature_names())