CODEWITHSUNDEEP
pythonlinuxpython-3.xfile-io
Torxed
Torxed

Reputation: 23480

for line in open(..) hangs unexpectedly mid file

When reading a file in the most basic manner like this:

with open('/tmp/abs/path/file.txt', 'rb') as fh:
    for line in fh:
        print('New line...'
        # calculations
        print('Last line of line-loop')
    print('Ended up outside') # <-- Never reaches this either, which is "fine".

The fileoperation hangs mid file with the last output being "Last line of line-loop" meaning the operation hangs on for line in fh for some reason.

The file-size is 5025728020 bytes
The file-position: 26957152 bytes

So i thought i'd use epoll() instead to Watch for blocking read by doing:

with open('/tmp/abs/path/file.txt', 'rb') as fh:
    watcher = select.epoll()
    watcher.register(fh.fileno(), select.EPOLLIN)

This fails however with operation not permitted, this has worked so many times in the past for some unknown reason both on files, sockets and stdout pipes, but not on this machine? Or have I just stuck lucky on some special disk-file Before?

Debian, v7.5 Python, v3.3.5

What puzzles me is that i have Three files, one being 1 billion bytes bigger, Another being just mere 3296 bytes. The bigger and the smaller file works perfectly with the same code, the one mentioned does not.

This puzzles me in so many ways that i don't even know where to begin.

Did a python3.3 -m trace --trace script.py and it says:

script.py(112):                for line in fh:
script.py(113):                    print(fh.tell())
124124124
script.py(112):                for line in fh:

And there it hangs.. forever..

Edit: It always hangs on the same place and tail can't get past this point either.
Running: tail -c+26956052 /file | head
Gives me one line, and one line only. It should give me more. Running just a tail /file hangs as well..

Tried file /file and it gave me:
... ASCII text, with very long lines

Martijn got me thinking, and suddenly i remember that "dd" is a thing sigh.
Doing dd if=file ibs=1 skip=26957150 count=100 gives me:

;
100+0 records in
0+1 records out
100 bytes (100 B) copied, 5.5e-05 s, 1.8MB/s

Something is definetly wrong with this file/hardware.

strace cat /file > /dev/null gives:

read(3, " some data that is correc"..., 32768) = 32768
write(3, " some data that is correc"..., 32768) = 32768
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0...", 32768) = 32768
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0...", 32768) = 32768
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0...", 32768) = 32768
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0...", 32768) = 32768
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0...", 32768) = 32768
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0...", 32768) = 32768
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0...", 1556) = 1556
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0...", 1556) = 1556
read(3, "", 32768)                = 0
close(3)                          = 0
close(1)                          = 0
close(2)                          = 0
exit_group(0)                     = ?

All help is apriciated, and i'll investigate the file transfer of this file.
The file is transfered via a scheduled 10min window via scp betweeen a LAN of 64MB/s transferrates. Something might be going wrong here, perhaps rsync will do a better job.

Upvotes: 2

Views: 1313

Answers (1)

MattH
MattH

Reputation: 38247

rsync has some advantages, automatic checksums provide additional protection (over just scp) against in-memory corruption, but as far as I know doesn't re-validate the destination file after writing.

If there was corruption during file transfer checksums would fail and you still should be able to read junk.

The fact that the read fails suggests filesystem or media corruption.

Upvotes: 1

Related Questions