CODEWITHSUNDEEP
pythonpandasscikit-learn
Lonewolf
Lonewolf

Reputation: 197

Having trouble reading Pandas dataframe with SciLearn Kit

I'm new to Python, and I am having trouble using SciLearn Kit on dataframes created using Pandas. Below is the code:

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib as plt
import json
%matplotlib inline

data = pd.read_json('C:/Users/Desktop/Machine Learning/yelp_academic_dataset_business.json', lines=True, orient='columns', encoding='utf-8')
dataframe = pd.DataFrame(data)

list(dataframe)
subset_data = dataframe.loc[(dataframe.city == 'Toronto')]
print(subset_data)
documents = subset_data.to_dict('records')

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer

no_features = 1000

# NMF is able to use tf-idf
tfidf_vectorizer = TfidfVectorizer(max_df=0.95, min_df=2, max_features=no_features, stop_words='english')
tfidf = tfidf_vectorizer.fit_transform(documents)
tfidf_feature_names = tfidf_vectorizer.get_feature_names()

# LDA can only use raw term counts for LDA because it is a probabilistic graphical model
tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2, max_features=no_features, stop_words='english')
tf = tf_vectorizer.fit_transform(documents)
tf_feature_names = tf_vectorizer.get_feature_names()

Below is the error I get.

AttributeError: 'dict' object has no attribute 'lower'

The dataset is available here : kaggle.com/yelp-dataset/yelp-dataset Dataset:yelp_academic_dataset_business.json

Any help will be appreciated. Thank you.

Upvotes: 2

Views: 118

Answers (1)

Venkatachalam
Venkatachalam

Reputation: 16966

As mentioned by @Jarad, You have to feed a list or series to tfidf_vectorizer. Hence, the fix to your issues is

tfidf = tfidf_vectorizer.fit_transform(subset_data[records])

Upvotes: 1

Related Questions