CODEWITHSUNDEEP
pythonaws-lambdagensimword2vecamazon-efs
Teja Rao
Teja Rao

Reputation: 21

unpacking error while loading word2vec pretrained model

I have a word2vec.model file. I am using this file in my lambda function. It was working fine. There was some change in the data file, so I had to retrain the model and use the updated word2vec.model

When I use the updated file, I am getting the below error.

{ "errorMessage": "invalid load key, '\\xef'.", 
    "errorType": "UnpicklingError", 
    "requestId": "3bdd63d9-39b7-4eee-81f1-493f2ab53749", 
    "stackTrace": [ " File \"/var/task/lambda_function.py\", line 173, in lambda_handler\n model = gensim.models.Word2Vec.load('word2vec.model')\n", " File \"/mnt/access/gensim/models/word2vec.py\", line 1953, in load\n model = super(Word2Vec, cls).load(*args, **kwargs)\n", " File \"/mnt/access/gensim/utils.py\", line 485, in load\n obj = unpickle(fname)\n", " File \"/mnt/access/gensim/utils.py\", line 1460, in unpickle\n return _pickle.load(f, encoding='latin1') # needed because loading from S3 doesn't support readline()\n" ] }

I tried loading the 'word2vec.model' file using:

import gensim model = gensim.models.Word2Vec.load('word2vec.model')

This worked fine with the older version of 'word2vec.model' But after I retrained it, it is giving me this error with the newer version of the 'word2vec.model'

I am working on AWS Lambda. The dependencides are installed in an EFS using EC2. The python package versions used in AWS Lambda are from the time when the model was trained for the first time.

The latest model is trained locally, using the latest version of gensim.

Could the reason be a change in version of gensim? But I don't remember the version number of gensim that was used in training the old 'word2vec.model' - which is working fine.

How do I resolve this? Any help is appreciated. Thanks!

Upvotes: 2

Views: 27

Answers (0)

Related Questions