Package requirements 'tensorflow~=2.10.0', 'transformers~=4.22.1', 'TorchCRF~=1.1.0' are not satisfied

Question

I am using PyCharm 2022.2.3 (Professional Edition), Python 3.11.0 (have any problem in compatible between this version of Python with PyTorch?) , Windows 11 x64.

Microsoft Windows [Version 10.0.22621.674]
(c) Microsoft Corporation. All rights reserved.

C:\Users\donhu>python
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

File requirement.txt

tensorflow~=2.10.0
numpy~=1.23.3
# torch==1.12.1
PyYAML~=6.0
pandas~=1.5.0
transformers~=4.22.1
TorchCRF~=1.1.0
tqdm~=4.64.1
seqeval~=1.2.2
viet-text-tools==0.1.1

Error

Package requirements 'tensorflow~=2.10.0', 'transformers~=4.22.1', 'TorchCRF~=1.1.0' are not satisfied

File train.py

import argparse
import yaml
import pandas as pd
import torch
from TorchCRF import CRF
import transformers
from data import Dataset
from engines import train_fn
import warnings

warnings.filterwarnings("ignore")
parser = argparse.ArgumentParser()
parser.add_argument("--data_file", type=str)
parser.add_argument("--hyps_file", type=str)
args = parser.parse_args()
data_file = yaml.load(open(args.data_file), Loader=yaml.FullLoader)
hyps_file = yaml.load(open(args.hyps_file), Loader=yaml.FullLoader)
train_loader = torch.utils.data.DataLoader(
    Dataset(
        df=pd.read_csv(data_file["train_df_path"]),
        tag_names=data_file["tag_names"],
        tokenizer=transformers.AutoTokenizer.from_pretrained(hyps_file["encoder"], use_fast=False),
    ),
    num_workers=hyps_file["num_workers"],
    batch_size=hyps_file["batch_size"],
    shuffle=True,
)
val_loader = torch.utils.data.DataLoader(
    Dataset(
        df=pd.read_csv(data_file["val_df_path"]),
        tag_names=data_file["tag_names"],
        tokenizer=transformers.AutoTokenizer.from_pretrained(hyps_file["encoder"], use_fast=False),
    ),
    num_workers=hyps_file["num_workers"],
    batch_size=hyps_file["batch_size"] * 2,
)
loaders = {
    "train": train_loader,
    "val": val_loader,
}
model = transformers.RobertaForTokenClassification.from_pretrained(hyps_file["encoder"],
                                                                   num_labels=data_file["num_tags"])
if hyps_file["use_crf"]:
    criterion = CRF(num_tags=data_file["num_tags"], batch_first=True)
else:
    criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=float(hyps_file["lr"]))
train_fn(
    loaders, model, torch.device(hyps_file["device"]), hyps_file["device_ids"],
    criterion,
    optimizer,
    epochs=hyps_file["epochs"],
    ckp_path="../ckps/{}.pt".format(hyps_file["encoder"].split("/")[-1]),
)

Message

ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch
WARNING: You are using pip version 21.3.1; however, version 22.3 is available.
You should consider upgrading via the 

Even update pip, not resolve problem

How to fix it?

Answer 1

For PyCharm, take a look at https://www.jetbrains.com/help/pycharm/managing-dependencies.html#populate_dependency_files

After you've set it up, you can click Install requirement.

Open a project with the requirements file specified, a notification bar is displayed on top of any Python or requirements file opened in Editor:

---

Beyond PyCharm you should also learn some bits of conda and different Python environment setup: https://docs.python-guide.org/dev/virtualenvs/

---

First read this https://docs.python-guide.org/dev/virtualenvs/ and https://conda.io/projects/conda/en/latest/user-guide/getting-started.html, then most probably, try conda before attempting the following.

And if you're still having issues after trying the above, then it's time to consider manually updating the (possibly stale) requirements.txt, e.g.

With the requirements:

tensorflow~=2.10.0
numpy~=1.23.3
# torch==1.12.1
PyYAML~=6.0
pandas~=1.5.0
transformers~=4.22.1
TorchCRF~=1.1.0
tqdm~=4.64.1
seqeval~=1.2.2
viet-text-tools==0.1.1

We must first kind of determine which one is lowest in the food chain that requires the most dependencies.

Generic Python libraries

tqdm~=4.64.1
PyYAML~=6.0

Data related:

pandas~=1.5.0
numpy~=1.23.3

Popular Machine Learning related:

tensorflow~=2.10.0
# torch==1.12.1
transformers~=4.22.1

Less popular machine learning tools:

seqeval~=1.2.2
TorchCRF~=1.1.0
viet-text-tools==0.1.1

• For the generic python tools, it should be very safe to just update them to the LATEST version possible

• For the data related and machine learning tools, the library that has the most dependencies would most probably be the transformers. In that case, you can safely update transformers to the LATEST STABLE version and that should give you appropriate matching numpy, pandas, tensorflow and pytorch versions.

Up till now our new requirements.txt would look like this:

tqdm
PyYAML
transformers

• Now, looking at the seqeval, it's a library that most probably work well with transformers, so adding it without specifying version should be okay

• Then we have the viet-text-tools==0.1.1 tagged to a specific versions without ~=

tqdm
PyYAML
transformers
seqeval
viet-text-tools==0.1.1

Then we're left with the biggest question mark, TorchCRF, we actually have two libraries that shares similar API:

• pip install pytorch-crf: https://pytorch-crf.readthedocs.io/en/stable/
• pip install TorchCRF: https://pypi.org/project/TorchCRF/

Most probably you want the latter one, so given that the torch related dependencies are handled by transformers, you can try just underspecifying the version for this.

So you final requirements.txt would look like this:

tqdm
PyYAML
transformers
seqeval
viet-text-tools==0.1.1
TorchCRF

After trying the above, do a pip freeze to take note of all the actual version of the dependencies that are installed.

Then edit the versions in your requirements.txt file accordingly, you should see something like this:

tqdm
PyYAML
transformers==4.20.1
seqeval==1.2.2
viet-text-tools==0.1.1
TorchCRF==1.1.0

(Note: You might see a different set of versions on your machine depending on your setuptools/pip versions)

---

Q: Do I always have to do these versioning manually?

Most probably not if you're using conda and pycharm integration https://www.jetbrains.com/help/pycharm/conda-support-creating-conda-virtual-environment.html

Q: Is it normal for environment setup to be so complicated?

Not really, but if we are setting up an environment setup for code that we're not familiar with, it's a little trial and error, esp. when the requirements.txt has many flexible compatibility ~=