CODEWITHSUNDEEP
pythonpandashdfstore
Jan van der Vegt
Jan van der Vegt

Reputation: 1511

Pandas HDFStore weird behaviour

I'm very confused by this, I have two scripts running from the same Notebook server. I have a HDFStore in the same folder, however for one script it looks empty and for the other it doesn't. Here is the same code run in both notebooks:

import os 
os.getcwd()

Return NB 1: 'E:\DoingDataScience\Identify\Scripts'

Return NB 2: 'E:\DoingDataScience\Identify\Scripts'

store = pd.HDFStore('data_test.h5')
store.items

Return NB 1: 

<bound method HDFStore.items of <class 'pandas.io.pytables.HDFStore'>
File path: data_test.h5
/test_database            frame        (shape->[3,5])>

Return NB 2:

<bound method HDFStore.items of <class 'pandas.io.pytables.HDFStore'>
File path: data_test.h5
Empty>

All the circumstances look the same. Does it keep track of the source of the Pandas DF? I thought it was maybe due the file already being opened by NB 1 because NB 1 created it. However killing NB 1 and restarting NB 2 didn't do anything either.

EDIT: Added os.stats:

NB 1 and NB 2 both:

nt.stat_result(st_mode=33206, st_ino=0L, st_dev=0L, st_nlink=0, 
st_uid=0, st_gid=0, st_size=1311400L, st_atime=1465072325L, 
st_mtime=1465149771L, st_ctime=1465072325L)

Update: Now after restarting I'm getting a new error:

HDF5ExtError: HDF5 error back trace

File "C:\aroot\work\hdf5-1.8.15-patch1\src\H5F.c", line 604, in H5Fopen unable to open file File "C:\aroot\work\hdf5-1.8.15-patch1\src\H5Fint.c", line 1085, in H5F_open unable to read superblock File "C:\aroot\work\hdf5-1.8.15-patch1\src\H5Fsuper.c", line 294, in H5F_super_read unable to load superblock File "C:\aroot\work\hdf5-1.8.15-patch1\src\H5AC.c", line 1320, in H5AC_protect H5C_protect() failed. File "C:\aroot\work\hdf5-1.8.15-patch1\src\H5C.c", line 3574, in H5C_protect can't load entry File "C:\aroot\work\hdf5-1.8.15-patch1\src\H5C.c", line 7954, in H5C_load_entry unable to load entry File "C:\aroot\work\hdf5-1.8.15-patch1\src\H5Fsuper_cache.c", line 476, in H5F_sblock_load truncated file: eof = 800, sblock->base_addr = 0, stored_eoa = 1181448

End of HDF5 error back trace

Unable to open/create file 'data_test.h5'

Upvotes: 0

Views: 1571

Answers (1)

MaxU - stand with Ukraine
MaxU - stand with Ukraine

Reputation: 210812

Most probably you've written your data 

/test_database            frame        (shape->[3,5])>

in the NB1's session and didn't close the store (which would save and flush your changes to file/disk)

So when you opened your h5 file in NB2, it wasn't yet flushed and therefore was empty.

Upvotes: 2

Related Questions