CODEWITHSUNDEEP
pythonpandasmemorydataframescikit-learn
Amal Kostali Targhi
Amal Kostali Targhi

Reputation: 903

pandas dataframe memory python

i want to transform a sparse matrix (156060x11780) to dataframe but i get a memory error this is my code

vect = TfidfVectorizer(sublinear_tf=True, analyzer='word', 
                       stop_words='english' , tokenizer=tokenize,
                       strip_accents = 'ascii') 

X = vect.fit_transform(df.pop('Phrase')).toarray()

for i, col in enumerate(vect.get_feature_names()):
    df[col] = X[:, i]

I have a problem in X = vect.fit_transform(df.pop('Phrase')).toarray(). How can i solve it?

Upvotes: 2

Views: 954

Answers (1)

MaxU - stand with Ukraine
MaxU - stand with Ukraine

Reputation: 210882

Try this:

from sklearn.feature_extraction.text import TfidfVectorizer
vect = TfidfVectorizer(sublinear_tf=True, analyzer='word', stop_words='english',
                       tokenizer=tokenize,
                       strip_accents='ascii',dtype=np.float16)
X = vect.fit_transform(df.pop('Phrase'))  # NOTE: `.toarray()` was removed


for i, col in enumerate(vect.get_feature_names()):
    df[col] = pd.SparseSeries(X[:, i].toarray().reshape(-1,), fill_value=0)

UPDATE: for Pandas 0.20+ we can construct SparseDataFrame directly from sparse arrays:

from sklearn.feature_extraction.text import TfidfVectorizer
vect = TfidfVectorizer(sublinear_tf=True, analyzer='word', stop_words='english',
                       tokenizer=tokenize,
                       strip_accents='ascii',dtype=np.float16)

df = pd.SparseDataFrame(vect.fit_transform(df.pop('Phrase')),
                        columns=vect.get_feature_names(),
                        index=df.index)

UPDATE from 2022-01-22 in modern versions of Pandas the pd.SparseDataFrame method has been deprecated, so please use pd.DataFrame.sparse.from_spmatrix() instead.

Upvotes: 3

Related Questions