CODEWITHSUNDEEP
pythonpandasmachine-learningscikit-learncategorical-data
James Wong
James Wong

Reputation: 1137

Feature preprocessing of both continuous and categorical variables (of integer type) with scikit-learn

The main goals are as follows:

  1. Apply StandardScaler to continuous variables

  2. Apply LabelEncoder and OnehotEncoder to categorical variables

The continuous variables need to be scaled, but at the same time, a couple of categorical variables are also of integer type. Applying StandardScaler would result in undesired effects.

On the flip side, the StandardScaler would scale the integer based categorical variables, which is also not what we want.

Since continuous variables and categorical ones are mixed in a single Pandas DataFrame, what's the recommended workflow to approach this kind of problem?

The best example to illustrate my point is the Kaggle Bike Sharing Demand dataset, where season and weather are integer categorical variables

Upvotes: 23

Views: 24205

Answers (2)

Kirushikesh
Kirushikesh

Reputation: 758

Checkout the ColumnTransformer in scikit-learn

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler,LabelBinarizer

numeric_columns=list(X.select_dtypes('float64').columns)
categorical_columns=list(X.select_dtypes('int64').columns)

pipeline=ColumnTransformer([
    ('num',StandardScaler(),numeric_columns),
    ('cat',LabelBinarizer(),categorical_columns),
])

new_X=pipeline.fit_transform(X)

Upvotes: 4

user1808924
user1808924

Reputation: 4926

Check out the sklearn_pandas.DataFrameMapper meta-transformer. Use it as the first step in your pipeline to perform column-wise data engineering operations:

mapper = DataFrameMapper(
  [(continuous_col, StandardScaler()) for continuous_col in continuous_cols] +
  [(categorical_col, LabelBinarizer()) for categorical_col in categorical_cols]
)
pipeline = Pipeline(
  [("mapper", mapper),
  ("estimator", estimator)]
)
pipeline.fit_transform(df, df["y"])

Also, you should be using sklearn.preprocessing.LabelBinarizer instead of a list of [LabelEncoder(), OneHotEncoder()].

Upvotes: 28

Related Questions