Reputation: 532

TypeError: h5py objects cannot be pickled

I am trying to run a PyTorch implementation of a code, which is supposed to work on SBD dataset.

The training labels are originally available in .bin file, which are then converted to HDF5 (.h5) files.

Upon running the algorithm, I get an error as: " TypeError: h5py objects cannot be pickled "

I think the error is stemming from torch.utils.data.DataLoader.

Any idea if I am missing any concept here? I read that pickling is generally not preferred but as of now, my dataset is in HDF5 format only.

For your reference, the error's stack trace is as follows:

  File "G:\My Drive\Debvrat - shared\Codes\CASENet PyTorch Implementations\SBD-lijiaman\main.py", line 130, in <module>
    main()

  File "G:\My Drive\Debvrat - shared\Codes\CASENet PyTorch Implementations\SBD-lijiaman\main.py", line 85, in main
    win_feats5, win_fusion, viz, global_step)

  File "G:\My Drive\Debvrat - shared\Codes\CASENet PyTorch Implementations\SBD-lijiaman\train_val\model_play.py", line 31, in train
    for i, (img, target) in enumerate(train_loader):

  File "C:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 819, in __iter__
    return _DataLoaderIter(self)

  File "C:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 560, in __init__
    w.start()

  File "C:\Anaconda3\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)

  File "C:\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)

  File "C:\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)

  File "C:\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)

  File "C:\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)

  File "C:\Anaconda3\lib\site-packages\h5py\_hl\base.py", line 308, in __getnewargs__
    raise TypeError("h5py objects cannot be pickled")

  TypeError: h5py objects cannot be pickled

I am using Conda version 4.8.2, Python 3.7.4, PyTorch 1.0.0 with Cuda 10.2.89

Thanks,

Upvotes: 1