python file read "int too large to convert to c long"

Question

On my (presumably 64-bit Windows, 64-bit 2.7 python) installation, the file read function is using a 4 byte c_long (signed long). I tested the base python file read function and I can't pass in an offset of more than the max signed long integer value (2,147,483,647). Not sure if this is due to a problem with my python installation, or if this is truly the max limit for reading from a file in python...

My test code is below:

import sys
import platform

inFileName = r'C:\Projects\Tampa\LASPY_EVLR\LAS_DATA\input\Large_LAS\20505.las'
bit32_offset_signedlong = 2147483647

print("python version: " + sys.version)
print("platform: " + str(platform.architecture()))
print("------------------------------")

fileref = open(inFileName, "r")

print("starting 32bit max read")
datpart_32bitmax = fileref.read(bit32_offset_signedlong)
print("------------------------------")

print("starting 32bit max plus one read")
datpart_32bitmaxplus1 = fileref.read(bit32_offset_signedlong + 1)
print("------------------------------")

This produces output like this:

python version: 2.7.12 |Continuum Analytics, Inc.| (default, Jun 29 2016, 11:07:13) [MSC v.1500 64 bit (AMD64)]
platform: ('64bit', 'WindowsPE')
------------------------------
starting 32bit max read
------------------------------
starting 32bit max plus one read
Traceback (most recent call last):
  File "C:\Projects\Tampa\LASPY_EVLR\check_clong.py", line 18, in <module>
    datpart_32bitmaxplus1 = fileref.read(bit32_offset_signedlong + 1)
OverflowError: Python int too large to convert to C long
Press any key to continue . . .

Is this normal? I thought that python could read an "unlimited" file size (solely limited by available RAM and OS bit size) as discussed here: Max size of a file Python can open?

I should make clear that this issue only shows up when using the offset parameter of the read method. I can read and write files larger than the 32-bit signed integer size, just when I try to read a portion of the file using the read offset parameter then the overflow error shows up. My end goal is to append some data near the tail end of the very large (6GB) file.

Is there something wrong with my python installation? If so, maybe there is something I can do to fix this issue...

Answer 1

This is happening because the function you are calling is layered on top of a C function that requires a 32-bit offset value. Python integers are not limited to this range, but C functions are.

Also note that the read would specify a read of up to 2GB if you ever managed that. Are you prepared to handle a 2GB string item if the file exceeds that length?