CODEWITHSUNDEEP
pythonpytorchhuggingface-transformershuggingface-tokenizershuggingface
Charlie Parker
Charlie Parker

Reputation: 5209

How to resolve the hugging face error ImportError: cannot import name 'is_tokenizers_available' from 'transformers.utils'?

I was trying to use the ViTT transfomer. I got the following error with code:

from pathlib import Path
import torchvision
from typing import Callable
root = Path("~/data/").expanduser()
# root = Path(".").expanduser()
train = torchvision.datasets.CIFAR100(root=root, train=True, download=True)
test = torchvision.datasets.CIFAR100(root=root, train=False, download=True)
img2tensor: Callable = torchvision.transforms.ToTensor()
from transformers import ViTFeatureExtractor
feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224-in21k")
x, y = train_ds[0]
print(f'{y=}')
print(f'{type(x)=}')
x = img2tensor(x)
x = x.unsqueeze(0)  # add batch size 1
out_cls: ImageClassifierOutput = model(x)
print(f'{out_cls.logits=}')

error

Files already downloaded and verified
Files already downloaded and verified
Traceback (most recent call last):
  File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 11, in <module>
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/transformers/__init__.py", line 30, in <module>
    from . import dependency_versions_check
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/transformers/dependency_versions_check.py", line 36, in <module>
    from .utils import is_tokenizers_available
ImportError: cannot import name 'is_tokenizers_available' from 'transformers.utils' (/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/transformers/utils/__init__.py)

I tried upgrading everything but it still failed. Upgrade commands:

/Users/brandomiranda/opt/anaconda3/envs/meta_learning/bin/python -m 
pip install --upgrade pip
pip install --upgrade pip
pip install --upgrade huggingface-hub

pip install --upgrade transformers
pip install --upgrade huggingface-hub
pip install --upgrade datasets

pip install --upgrade tokenizers

pip install pytorch-transformers

pip install --upgrade torch
pip install --upgrade torchvision
pip install --upgrade torchtext
pip install --upgrade torchaudio

# pip install --upgrade torchmeta
pip uninstall torchmeta

Why and how to fix it?

Pip list:

(meta_learning) ❯ pip list
Package                                           Version    Editable project location
------------------------------------------------- ---------- ------------------------------------------------------------------------------
absl-py                                           1.0.0
aiohttp                                           3.8.1
aiosignal                                         1.2.0
antlr4-python3-runtime                            4.8
argcomplete                                       2.0.0
async-timeout                                     4.0.1
attrs                                             21.4.0
automl-meta-learning                              0.1.0      /Users/brandomiranda/automl-meta-learning/automl-proj-src
bcj-cffi                                          0.5.1
boto                                              2.49.0
boto3                                             1.24.85
botocore                                          1.27.85
Bottleneck                                        1.3.4
Brotli                                            1.0.9
brotlicffi                                        1.0.9.2
brotlipy                                          0.7.0
cachetools                                        4.2.4
certifi                                           2022.9.14
cffi                                              1.15.1
charset-normalizer                                2.0.9
cherry-rl                                         0.1.4
click                                             8.0.3
cloudpickle                                       2.0.0
colorama                                          0.4.4
configparser                                      5.2.0
conllu                                            4.4.1
crcmod                                            1.7
cryptography                                      37.0.1
cycler                                            0.11.0
Cython                                            0.29.25
dataclasses                                       0.6
datasets                                          2.5.1
dill                                              0.3.4
diversity-for-predictive-success-of-meta-learning 0.0.1      /Users/brandomiranda/diversity-for-predictive-success-of-meta-learning/div_src
docker-pycreds                                    0.4.0
editdistance                                      0.6.0
et-xmlfile                                        1.1.0
fairseq                                           0.10.0
fastcluster                                       1.2.4
fasteners                                         0.17.3
filelock                                          3.6.0
fonttools                                         4.28.3
frozenlist                                        1.2.0
fsspec                                            2022.7.1
gcs-oauth2-boto-plugin                            3.0
gitdb                                             4.0.9
GitPython                                         3.1.24
google-apitools                                   0.5.32
google-auth                                       2.3.3
google-auth-oauthlib                              0.4.6
google-reauth                                     0.1.1
grpcio                                            1.42.0
gsutil                                            5.6
gym                                               0.21.0
h5py                                              3.6.0
higher                                            0.2.1
httplib2                                          0.20.4
huggingface-hub                                   0.10.0
hydra-core                                        1.1.1
idna                                              3.3
importlib-metadata                                4.11.3
jmespath                                          1.0.1
joblib                                            1.1.0
kiwisolver                                        1.3.2
lark-parser                                       0.12.0
learn2learn                                       0.1.7
lxml                                              4.8.0
Markdown                                          3.3.6
matplotlib                                        3.5.1
mkl-fft                                           1.3.1
mkl-random                                        1.2.2
mkl-service                                       2.4.0
monotonic                                         1.6
multidict                                         5.2.0
multiprocess                                      0.70.12.2
multivolumefile                                   0.2.3
munkres                                           1.1.4
networkx                                          2.6.3
numexpr                                           2.8.1
numpy                                             1.21.5
oauth2client                                      4.1.3
oauthlib                                          3.1.1
omegaconf                                         2.1.1
openpyxl                                          3.0.10
ordered-set                                       4.0.2
packaging                                         21.3
pandas                                            1.4.2
pathtools                                         0.1.2
Pillow                                            9.0.1
pip                                               22.2.2
plotly                                            5.4.0
portalocker                                       2.3.2
progressbar2                                      3.55.0
promise                                           2.3
protobuf                                          3.19.1
psutil                                            5.8.0
py7zr                                             0.16.1
pyarrow                                           9.0.0
pyasn1                                            0.4.8
pyasn1-modules                                    0.2.8
pycparser                                         2.21
pycryptodomex                                     3.15.0
pyOpenSSL                                         22.0.0
pyparsing                                         3.0.6
pyppmd                                            0.16.1
PySocks                                           1.7.1
python-dateutil                                   2.8.2
python-utils                                      2.5.6
pytorch-transformers                              1.2.0
pytz                                              2021.3
pyu2f                                             0.1.5
PyYAML                                            6.0
pyzstd                                            0.14.4
qpth                                              0.0.15
regex                                             2021.11.10
requests                                          2.28.1
requests-oauthlib                                 1.3.0
responses                                         0.18.0
retry-decorator                                   1.1.1
rsa                                               4.7.2
s3transfer                                        0.6.0
sacrebleu                                         2.0.0
sacremoses                                        0.0.46
scikit-learn                                      1.0.1
scipy                                             1.7.3
seaborn                                           0.11.2
sentencepiece                                     0.1.97
sentry-sdk                                        1.5.1
setproctitle                                      1.2.2
setuptools                                        58.0.4
shortuuid                                         1.0.8
six                                               1.16.0
sklearn                                           0.0
smmap                                             5.0.0
subprocess32                                      3.5.4
tabulate                                          0.8.9
tenacity                                          8.0.1
tensorboard                                       2.7.0
tensorboard-data-server                           0.6.1
tensorboard-plugin-wit                            1.8.0
termcolor                                         1.1.0
texttable                                         1.6.4
threadpoolctl                                     3.0.0
tokenizers                                        0.13.0
torch                                             1.12.1
torchaudio                                        0.12.1
torchtext                                         0.13.1
torchvision                                       0.13.1
tornado                                           6.1
tqdm                                              4.62.3
transformers                                      4.22.2
typing_extensions                                 4.3.0
ultimate-anatome                                  0.1.1      /Users/brandomiranda/ultimate-anatome
ultimate-aws-cv-task2vec                          0.0.1      /Users/brandomiranda/ultimate-aws-cv-task2vec
ultimate-utils                                    0.6.1      /Users/brandomiranda/ultimate-utils/ultimate-utils-proj-src
urllib3                                           1.26.11
wandb                                             0.13.3
Werkzeug                                          2.0.2
wheel                                             0.37.0
xxhash                                            2.0.2
yarl                                              1.8.1
yaspin                                            2.1.0
zipp                                              3.8.0

hugging face (HF) related gitissues:

Upvotes: 0

Views: 20780

Answers (2)

gaspar
gaspar

Reputation: 59

If the model was defined, I would argue that it looks like an anaconda problem, I suggest using a different virtual environment to test this, for example, pipenv. Another option would be to use the timm library which also has models for image classification.

Upvotes: 0

Bob
Bob

Reputation: 14654

You did not give the model in your question.

Using google colab I could easily run the models openai/clip-vit-base-patch32

from PIL import Image
import requests

from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["a photo of a cat", "a photo of a dog"], 
                   images=image, 
                   return_tensors="pt", 
                   padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities

And also google/vit-base-patch16-224-in21k

from transformers import ViTFeatureExtractor, ViTModel
import torch

feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224-in21k")
model = ViTModel.from_pretrained("google/vit-base-patch16-224-in21k")

inputs = feature_extractor(image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state
list(last_hidden_states.shape)

This last example is a good starting point if you want to use this in your own task, as you can pass the last_hidden_state output to a custom model you will train.

However if you try these models directly cifar-100 dataset, you will run into a shape mismatch problem. The models require 244 x 244 images, but cifar dataset is of 32 x 32 images.

Upvotes: 1

Related Questions