ValueError: X has 500000 features, but ExtraTreeClassifier is expecting 7 features as input

Question

I am trying to develop a UI for a machine learning model that i implemented with the extratees classifier.

Below code shows how i exported the model after training to use in the UI. The prediction is done using the is_attributed column.

import numpy as np
import pandas as pd
from collections import Counter
import datetime
from sklearn.model_selection import train_test_split
from sklearn.model_selection import RepeatedStratifiedKFold
import gc

import warnings
warnings.simplefilter('ignore')

df = pd.read_csv('../cleaned_train.csv', index_col=0)

df['click_time'] = pd.to_datetime(df['click_time'])

df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 10000000 entries, 0 to 9999999
Data columns (total 9 columns):
 #   Column         Dtype         
---  ------         -----         
 0   ip             int64         
 1   app            int64         
 2   device         int64         
 3   os             int64         
 4   channel        int64         
 5   click_time     datetime64[ns]
 6   is_attributed  int64         
 7   hour           int64         
 8   day            int64         
dtypes: datetime64[ns](1), int64(8)
memory usage: 762.9 MB

X= df.drop(columns=['is_attributed', 'click_time'])
y= df['is_attributed']

#Undersample data
from imblearn.under_sampling import RandomUnderSampler

rus = RandomUnderSampler() 
X_res, y_res = rus.fit_resample(X, y)

X_train, X_test, y_train, y_test = train_test_split(X_res, y_res, test_size = 0.33, 
random_state = 0)

from sklearn.ensemble import ExtraTreesClassifier
from sklearn.model_selection import GridSearchCV
import pickle

# ExtraTreesClassifier
ec = ExtraTreesClassifier(max_depth=None, n_estimators=50)
ec.fit(X_train, y_train)

y_predec=ec.predict(X_test)
pickle.dump(gsec,open('model.pkl','wb'))

when i try to print this print(gsec.predict(X_test)) i get the results as [1 1 0 ... 1 1 0]

The issue comes when i try to develop a UI with flask. I imported the model in flask and attempted to predict. Below is the code for that.

# importing necessary libraries and functions
import numpy as np
import pandas as pd
from flask import Flask, request, jsonify, render_template, make_response
from werkzeug.utils import secure_filename
from werkzeug.datastructures import  FileStorage
import pickle
import io
from io import StringIO
import csv

app = Flask(__name__) #Initialize the flask App

@app.route('/') # Homepage
def home():
return render_template('index.html')


@app.route('/predict',methods=['GET', 'POST'])
def predict():
'''
For rendering results on HTML GUI
'''

# retrieving values from form
if request.method == 'POST':
  f = request.files['data_file']
  if not f:
    return "No file"

stream = io.StringIO(f.stream.read().decode("UTF8"), newline=None)
csv_input = csv.reader(stream)
# print(csv_input)
for row in csv_input:
    print(row)

stream.seek(0)
result = stream.read()

df = pd.read_csv('newcleaned_test.csv')
attribute = df['is_attributed']
ip = df['ip']

print (attribute)

# load the model from disk
loaded_model = pickle.load(open('model.pkl', 'rb'))
prediction = loaded_model.predict([attribute])

print (prediction)
return 'prediction'



if __name__ == "__main__":
app.run(debug=True)

When trying run the above code,

ValueError: X has 500000 features, but ExtraTreeClassifier is expecting 7 features as input.

is shown in my browser. (The data file i'm using has 500000 data with 7 columns). Why is this error thrown when i trained the model using one column?

Answer 1

You have a few misunderstandings here.

Firstly, as from code, you can see that model is trained on 7 columns as inputs [ip, app, device, os, channel, hour, day]. And the model is trained to predict values from is_attributed column. So feed a model list with 7 values -> receive 1 value as output. And this value seems to be 0 or 1 depends on input 7 values.

Secondly, we can proceed now to the Flask part. Basically, what you do here is that you load dataframe and select one column (attribute = df['is_attributed']). IF you have dataframe with 50000 rows and you select one column it means that you select 50000 values! And then you tried to send this to model, which wants exactly 7 values as inputs. As from my perspective, it looks like that you want to run model on each row on test dataframe.

To do that you need:

1. load test dataframe;
2. check that you have only 7 columns ([ip, app, device, os, channel, hour, day]) in dataframe. If you have more columns, remove all other;
3. go through each row in dataframe (each from total of 50000);
4. run model using 7 values from row as inputs;
5. model's output append to a python list;
6. after 50000 runs you will have python list with 50000 values;
7. return this list.