Reputation: 557
I was running isolation forest trying to apply it on a 10049972 rows x 19 columns database, but after 2 hours of running I got the following error. I really don't understand why did I get it, nor how do I resolve it?
Code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.ensemble import IsolationForest
df = pd.read_csv('D:\\Project\\database\\4-Final\\Final After.csv',low_memory=True)
iForest = IsolationForest(behaviour='new', n_estimators=80, contamination='auto' , max_samples=150)
df['anomaly'] = iForest.fit_predict(df.values.reshape(-1,1))
df=df.drop(df['anomaly'==-1],inplace=True)
df.to_csv('D:\\Project\\database\\4-Final\\IF TEST.csv', index=False)
and the error is:
ValueError Traceback (most recent call last)
<ipython-input-1-fc55c8b1f328> in <module>
16
17
---> 18 df['anomaly'] = iForest.fit_predict(df.values.reshape(-1,1))
19
20
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in __setitem__(self, key, value)
3368 else:
3369 # set column
-> 3370 self._set_item(key, value)
3371
3372 def _setitem_slice(self, key, value):
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in _set_item(self, key, value)
3443
3444 self._ensure_valid_index(value)
-> 3445 value = self._sanitize_column(key, value)
3446 NDFrame._set_item(self, key, value)
3447
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in _sanitize_column(self, key, value, broadcast)
3628
3629 # turn me into an ndarray
-> 3630 value = sanitize_index(value, self.index, copy=False)
3631 if not isinstance(value, (np.ndarray, Index)):
3632 if isinstance(value, list) and len(value) > 0:
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in sanitize_index(data, index, copy)
517
518 if len(data) != len(index):
--> 519 raise ValueError('Length of values does not match length of index')
520
521 if isinstance(data, ABCIndexClass) and not copy:
ValueError: Length of values does not match length of index
Thank you.
Upvotes: 0
Views: 208
Reputation: 56
I think the problem might be with
df.values.reshape(-1,1)
Look at this example
df = pd.DataFrame([(.2, .3), (.0, .6), (.6, .0), (.2, .1)], columns=['dogs', 'cats'])
df
dogs cats
0 0.2 0.3
1 0.0 0.6
2 0.6 0.0
3 0.2 0.1
df.values.reshape(-1,1)
array([[0.2],
[0.3],
[0. ],
[0.6],
[0.6],
[0. ],
[0.2],
[0.1]])
So you end up providing a shape (n_samples*n_feature, 1)
vector to the fit_predict
and you are plugging back the resulting shape (n_samples*n_feature,)
as a column to a df
with shape (n_samples,n_feature)
. There's a mismatch between the number of rows.
Upvotes: 1