CODEWITHSUNDEEP
pythonhuggingfacehuggingface-datasets
Yana
Yana

Reputation: 985

How to save custom dataset in local folder

I have created a custom huggingface dataset, containing images and ground truth data coming from json lines file. I want to save it to a local folder and be able to use it as is by loading it after to other notebooks. I did not find out how this can happen.

DatasetDict({
    train: Dataset({
        features: ['image', 'id', 'ground_truth'],
        num_rows: 7
    })
    test: Dataset({
        features: ['image', 'id', 'ground_truth'],
        num_rows: 4
    })
})

Upvotes: 1

Views: 3571

Answers (2)

tomasborrella
tomasborrella

Reputation: 530

According to huggingface documentacion you can use save_to_disk which "saves a dataset to a dataset directory, or in a filesystem".

Example:

ds.save_to_disk("path/to/dataset/dir")

Upvotes: 3

Yana
Yana

Reputation: 985

I have found a very easy way to do it in this documentation:

dataset.save_to_disk("path-to-dataset")

Upvotes: 0

Related Questions