CODEWITHSUNDEEP
pythonnumpyscipy
brorfred
brorfred

Reputation: 492

Max limit for file size with np.fromfile?

If there a max file size for np.fromfile()? I'm trying to read a 48GB file, with about 2.1e9 records (each record has 5 values).

It works when I limit the count to 8e7:

fromfile(filename, dtp, int(8e7))
Out[69]: 
array([(1, 244025.0, 1.1666666269302368, 360.1666564941406, 50.0),
       (2, 244025.0, 1.1666666269302368, 360.5, 50.0),
       (3, 244025.0, 1.1666666269302368, 360.8333435058594, 50.0), ...,
       (255138, 244131.0, 1128.9346923828125, 461.38494873046875, 49.5),
       (255139, 244131.0, 1143.77783203125, 473.2532958984375, 49.5),
       (255140, 244131.0, 1150.4803466796875, 464.6799011230469, 49.5)], 
      dtype=[('ntrac', '<i4'), ('ints', '<f8'), ('x', '<f4'), ('y', '<f4'), ('z', '<f4')])

But all values are read as zeros when the count is set to 9e7:

In [70]: fromfile(filename, dtp, int(9e7))
Out[70]: 
array([(0, 0.0, 0.0, 0.0, 0.0), (0, 0.0, 0.0, 0.0, 0.0),
       (0, 0.0, 0.0, 0.0, 0.0), ..., (0, 0.0, 0.0, 0.0, 0.0),
       (0, 0.0, 0.0, 0.0, 0.0), (0, 0.0, 0.0, 0.0, 0.0)], 
      dtype=[('ntrac', '<i4'), ('ints', '<f8'), ('x', '<f4'), ('y', '<f4'), ('z', '<f4')])

Upvotes: 3

Views: 1439

Answers (2)

Saulius Lukauskas
Saulius Lukauskas

Reputation: 362

It looks like it is indeed a bug with Mac OS X.

See the following bug in numpy: https://github.com/numpy/numpy/issues/2806

I have a proposed fix in my fork of the branch, would help for someone to verify it, see: https://github.com/sauliusl/numpy/tree/bug_large_save

Upvotes: 2

pv.
pv.

Reputation: 35125

It's possible it's a bug in Mac OSX's fread() routine. The fwrite() routine does have a similar bug: https://github.com/numpy/numpy/issues/574

You can maybe check this by writing a simple C test program (such as the one attached to the ticket there), and then complain to Apple...

The work-around is to read the file in chunks small enough.

EDIT: read the bug report too fast --- the fwrite() bug is not exactly similar, but I think an OS bug cannot be ruled out...

Upvotes: 2

Related Questions