CODEWITHSUNDEEP
numpyhuggingface-transformershuggingface-datasets
user1680859
user1680859

Reputation: 1194

HuggingFace Dataset - pyarrow.lib.ArrowMemoryError: realloc of size failed

I am trying to use Hugginface Datasets for speech recognition using transformers, where I have pairs of text/audio. I am creating a Dataframe without problem with these two lists:

d = pd.DataFrame.from_dict({"audio": ts_audios, "sentence": ts_sent})

But when trying to wrap this to a Dataset (from Hugginface datasets):

ds=Dataset.from_pandas(d)

it gives:

pyarrow.lib.ArrowMemoryError: realloc of size 4294967296 failed

The problem is because of the audios list, that looks like this:

[array([ 1.3715802e-05,  1.3041631e-05, -1.5017368e-06, ...,
       -1.1172481e-01, -1.2214723e-01,  0.0000000e+00], dtype=float32), array([-0.06073862, -0.12271373, -0.11600843, ..., -0.11915235,
       -0.13458692,  0.        ], dtype=float32), array([-0.07074431, -0.12263235, -0.1065825 , ..., -0.10845864,
       -0.12171803,  0.        ], dtype=float32), array([-0.02499148, -0.04160473, -0.03867628, ..., -0.01881211,
       -0.02035856,  0.        ], dtype=float32), array([-0.18304674, -0.03917564, -0.030768  , ..., -0.11494933,
       -0.112398  , -0.12073436], dtype=float32) .....]

I must use the Dataset format if I want to use transformers package from Huggingface. Any idea how can I solve this issue?

Upvotes: 2

Views: 8312

Answers (1)

Frank Bao
Frank Bao

Reputation: 31

Note that 4294967296 bytes are exactly 4GB of memory. Huggingface datasets will load the data in a lazy mode, so it may solve the memory problem. You can try to use datasets.audio to solve your problem. Here's an example from the official tutorial:

from datasets import load_dataset, load_metric, Audio
common_voice = load_dataset("common_voice", "tr", split="train")
common_voice[0]["audio"]

{'array': array([ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00, ...,
    -8.8930130e-05, -3.8027763e-05, -2.9146671e-05], dtype=float32),
'path': '/root/.cache/huggingface/datasets/downloads/extracted/05be0c29807a73c9b099873d2f5975dae6d05e9f7d577458a2466ecb9a2b0c6b/cv-corpus-6.1-2020-12-11/tr/clips/common_voice_tr_21921195.mp3',
'sampling_rate': 48000}

Or, if you've got the path to the audio file, you can also use cast_column to load the audio files.

my_audio_dataset = load_dataset("path_to_your_dataset",split="train")
my_audio_dataset = my_audio_dataset.cast_column("the_column_name_for_audio_files_path", Audio())

Upvotes: 1

Related Questions