CODEWITHSUNDEEP
pythonmachine-learningscikit-learn
vishak bharadwaj
vishak bharadwaj

Reputation: 71

Error creating a custom transformer in sklearn - takes 2 positional arguments but 3 were given

I'm trying to create a Pipeline using custom transfomers for a generic dataset. Here is my first transformer. Given a column name, it breaks that datetime column into further columns.

class DatePartTransformer:

    def __init__(self,fldname):
        self.fldname = fldname

    def fit(self):
        return self 

    def transform(self):
        return self

    def fit_transform(self,df, drop=True, time=False, errors='raise'):
        fld = df[self.fldname]
        fld_dtype = fld.dtype
        if isinstance(fld_dtype, pd.core.dtypes.dtypes.DatetimeTZDtype):
            fld_dtype = np.datetime64

        if not np.issubdtype(fld_dtype, np.datetime64):
            df[self.fldname] = fld = pd.to_datetime(fld, infer_datetime_format=True, errors=errors)
        targ_pre = re.sub('[Dd]ate$', '', self.fldname)
        attr = ['Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear',
            'Is_month_end', 'Is_month_start', 'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start']
        if time: attr = attr + ['Hour', 'Minute', 'Second']
        for n in attr: df[targ_pre + n] = getattr(fld.dt, n.lower())
        df[targ_pre + 'Elapsed'] = fld.astype(np.int64) // 10 ** 9
        if drop: df.drop(self.fldname, axis=1, inplace=True)

        return df

and here is my second 

from pandas.api.types import is_string_dtype

class TrainCats:

    def __init__(self):
        pass

    def fit(self):
        return self

    def transform(self):
        return self

    def fit_transform(self,df):

        for n,c in df.items():
            if is_string_dtype(c): 
                df[n] = c.astype('category').cat.as_ordered()
        return df

I plan to write more.

Here is the pipeline. 

pipeline = Pipeline([
 ('imputer',DatePartTransformer('date')),
 ('cats',TrainCats())
])

df = pipeline.fit_transform(df_raw)

When I run the pipeline, I get this error

TypeError                                 Traceback (most recent call last)
<ipython-input-36-36154d1b45b5> in <module>
      4 ])
      5 
----> 6 df = pipeline.fit_transform(df_raw)

c:\users\vishak~1\desktop\env\ml\lib\site-packages\sklearn\pipeline.py in fit_transform(self, X, y, **fit_params)
    391                 return Xt
    392             if hasattr(last_step, 'fit_transform'):
--> 393                 return last_step.fit_transform(Xt, y, **fit_params)
    394             else:
    395                 return last_step.fit(Xt, y, **fit_params).transform(Xt)

TypeError: fit_transform() takes 2 positional arguments but 3 were given

Aurélien Géron's book says this is how pipelines work. I'm not able to find my error.

Upvotes: 0

Views: 2341

Answers (1)

Proko
Proko

Reputation: 2011

If you look at the source code of Pipeline you will see that it requires for every transformer to take 2 positional arguments, that is X and y (apart from self) when using fit_transform method. This is exactly this line:

                 return last_step.fit_transform(Xt, y, **fit_params)

So the method declaration of fit_transform of your transformer must have 2 positional arguments. To fix it, all you need to do is to provide second dummy argument to your TrainCats fit_transform method like this:

    def fit_transform(self,df, y=None):

        for n,c in df.items():
            if is_string_dtype(c): 
                df[n] = c.astype('category').cat.as_ordered()
        return df

This will reduce your error but there is one more vulnerability. Although your fit_transform in DatePartTransformer takes more than 1 argument, due to the pipline assumption, your drop argument will be overriden with None or actual y from other transformer. If you expect to work only on inputs and not labels, you need to add this dummy argument to DatePartTransformer as well:

     def fit_transform(self,df, y=None, drop=True, time=False, errors='raise'):
        ...

Upvotes: 1

Related Questions