CODEWITHSUNDEEP
pythonnlphuggingface-datasetshuggingface
TMN
TMN

Reputation: 153

Huggingface datasets ValueError

I am trying to load a dataset from huggingface organization, but I am getting the following error:

ValueError: Couldn't cast string
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 686
to
{'text': Value(dtype='string', id=None)}
because column names don't match

I used the following lines of code to load the dataset:

from datasets import load_dataset
dataset = load_dataset("datasetFile", use_auth_token=True)

Pleases note the dataset version = (2.0.0), I changed it to 1.18.2 but it did not work.

Is there any way to fix this error?

Upvotes: 0

Views: 8362

Answers (2)

Serge Chastel
Serge Chastel

Reputation: 170

According to https://github.com/huggingface/datasets/issues/3700#issuecomment-1035400186, you actually want to use load_from_disk:

from datasets import load_from_disk
dataset = load_from_disk("datasetFile")

Upvotes: 2

TMN
TMN

Reputation: 153

I solved this error by streaming the dataset.

from datasets import load_dataset
dataset = load_dataset("datasetFile", use_auth_token=True, streaming= True)

Upvotes: 2

Related Questions