CODEWITHSUNDEEP
scikit-learnpipeline
Saurabh Jain
Saurabh Jain

Reputation: 1287

how to use output of sklearn pipeline elements

I have three features:

feature_one -> number of tokens in the given sentence.  
feature_two -> number of verbs in the given sentence.  
feature_three -> number of tokens - number of verbs in the given sentence.  
(feature_one - feature_two)

I have written custom transformers for feature_one and feature_two and want to written custom transformer for feature_three such that I can use result of feature_one and feature_two by running pipeline as: 

Pipeline([
        #input to feature_one and feature_two is list of sentences.
        ("feature", FeatureUnion([
            ("feature_one", feature_one_transformer()),
            ("feature_two", feature_two_transformer())          
        ])),

        ("feature_three", feature_three_transformer())    
])

feature_one_transformer: 

class feature_one_transformer(BaseEstimator, TransformerMixin):

    def __init__(self):
        pass

    def fit(self, x, y):
        return self

    def transform(self, sentence_list):
        number_of_tokens_in_sentence_list = list()

        for sentence in sentence_list:
            number_of_tokens = compute_number_of_tokens

            number_of_tokens_in_sentence_lista.append(number_of_tokens)

        return pandas.DataFrame(number_of_tokens_in_sentence_list)

feature_two_transformer: 

class feature_two_transformer(BaseEstimator, TransformerMixin):
    def __init__(self):
        pass

    def fit(self, x, y):
        return self

    def transform(self, sentence_list):
        number_of_verbs_in_sentence_list = list()

        for sentence in sentence_list:
            number_of_verbs = compute_number_of_verbs_in_sentence

            number_of_verbs_in_sentence_lista.append(number_of_verbs)

        return pandas.DataFrame(number_of_verbs_in_sentence_list)

Can somebody tell me how should I write custom transformer for feature_three and how to use in pipeline so that I can use result of feature_one and feature_two transformers. Thank you.

Upvotes: 0

Views: 814

Answers (1)

Oriol Mirosa
Oriol Mirosa

Reputation: 2826

It's not clear to me why you would want to make this so complicated. I would just use one transformer that does everything you want. Something like this:

class features_transformer(BaseEstimator, TransformerMixin):

    def __init__(self, variable):
        self.variable = variable

    def fit(self, X):
        return self

    def transform(self, X):
        X['number_of_tokens'] = X[self.variable].apply(lambda cell: compute_number_of_tokens(cell))
        X['number_of_verbs'] = X[self.variable].apply(lambda cell: compute_number_of_verbs(cell))
        X['tokens_minus_verbs'] = X['number_of_tokens'] - X['number_of_verbs']

        return X

new_X = features_transformer('sentences').fit_transform(X)

Upvotes: 1

Related Questions