CODEWITHSUNDEEP
pythonscikit-learnpicklegridsearchcv
hafiz031
hafiz031

Reputation: 2710

AttributeError: Can't get attribute 'tokenizer' on <module '__main__'>

I trained a logistic regression model on textual data and saved the model using pickle. But for testing when I try to load the model I got the error mentioned in the title while executing the following line:

model = pickle.load(open("sentiment.model", "rb"))

Following is the code used for saving the model:

import pickle

print("[INFO] saving Model...")
f = open('sentiment.model', "wb")

# first I saved the best_estimator_
f.write(pickle.dumps(gs_lr_tfidf.best_estimator_))

# but again I saved the model completely without mentioning any attribute i.e: 
# f.write(pickle.dumps(gs_lr_tfidf))
# but none of them helped and I got the same error

f.close()
print("[INFO] Model saved!")

This error doesn't show up when I load the model in the same notebook just after finishing the training process (in the same runtime). But this error occurs when I try to load the model separately in different runtime even if the model loader code is the same. Why this is happening?

Upvotes: 2

Views: 2563

Answers (1)

c yipkwun
c yipkwun

Reputation: 41

I think the problems is from the behaviour of pickle, as what @hafiz031 said, it's normal when run the same code in the file. So short answer is you need to import tokenizer(from whatever lib you use) before you load the model

For people who know chinese, you can go to this CSDN link for more info.

For people who don't know chinese, sorry for my bad English and I'll try my best to explain.

The documentation says:

pickle.loads(data, /, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=None)

Return the reconstituted object hierarchy of the pickled representation data of an object. data must be a bytes-like object.

There is an implicit requirement if you use pickle.loads, the object hierarchy must be declared before you load it. Intuitively you can think as you bring USD to north pole and you want to exchange USD to fish with a penguin. As they don't have the concept what is money, they won't make the deal. Same as pickle, if you haven't import tokenizer before, after pickle loads the bytes back to tokenizer, they don't know what is 'tokenizer' and return error to you. Thats why your code works in training file but fail when you loads the model in a different file.

in my case, I just import an extra lib.

# import your own lib
import pickle

import nltk.tokenizer
import genism
import sklearn
#...

model = pickle.load(open("sentiment.model", "rb"))
#model.predict()

Upvotes: 1

Related Questions