CODEWITHSUNDEEP
pythonscikit-learn
backtrack
backtrack

Reputation: 8154

Error in FeatureUnion Sklearn Pipeline

I have the following dataframe:

ID Text 
1  qwerty
2  asdfgh

I am trying to create md5 hash for Text field and remove ID field from the dataframe above. To achieve that i have created a simple pipeline with custom transformers from sklearn.

Here is the code I have used: 

class cust_txt_col(sklearn.base.BaseEstimator, sklearn.base.TransformerMixin):
    def __init__(self, key):
        self.key = key
    def fit(self, x, y=None):
        return self

    def hash_generate(self, txt):

        m = hashlib.md5()
        text = str(txt)
        long_text = ' '.join(text.split())
        m.update(long_text.encode('utf-8'))
        text_hash= m.hexdigest()
        return text_hash

    def transform(self, x):
        return x[self.key].apply(lambda  z: self.hash_generate(z)).values

class cust_regression_vals(sklearn.base.BaseEstimator, sklearn.base.TransformerMixin):
    def fit(self, x, y=None):
        return self
    def transform(self, x):
        x = x.drop(['Gene', 'Variation','ID','Text'], axis=1)
        return x.values

fp = pipeline.Pipeline([

 ('union', pipeline.FeatureUnion([
        ('hash', cust_txt_col('Text')), # can pass in either a pipeline
        ('normalized', cust_regression_vals()) # or a transformer
    ]))
])

When I run this I receive the follwoing error: 

ValueError: all the input arrays must have same number of dimensions

Can you, please, tell me what is wrong with my code?

if i run the classes one by one :

for cust_txt_col i got below o/p

['3e909f222a1e06098ec7ca1ea7e84540' '1691bdba3b75df145169e0501369fce3'
 '1691bdba3b75df145169e0501369fce3' ..., 'e11ec9863aaeb93f77a231319021e14d'
 '851c517b2af0a46cb9bc9373b748b6ff' '0ffe46fc75d21a5347b1f1a5a84526ad']

for cust_regression_vals i got below o/p

[[qwerty],
  [asdfgh]]

Upvotes: 4

Views: 585

Answers (1)

joeln
joeln

Reputation: 3643

cust_txt_col is returning a 1d array. FeatureUnion demands that each constituent transformer returns a 2d array.

Upvotes: 3

Related Questions