CODEWITHSUNDEEP
pythonfilesplitbyte
ByteFlowr
ByteFlowr

Reputation: 500

How can I read a file in 9 megabyte blocks in python?

After I gzipped a file:

fin = open("foo.mp4", "rb")
fout = gzip.open("data.tmp", "wb")
fout.write(fin.read())
fout.close()
fin.close()

I want to load the file (the file could have any size) into memory in 9 megabyte blocks.

Each block should be 9mb, the last one might be smaller. I need this size in order to upload the data to a POST-Endpoint which only acceptes <= 9mb of file size.

Any idea how to read the file in without having to do subprocess calls to split?

Upvotes: 1

Views: 193

Answers (2)

Kenly
Kenly

Reputation: 26698

You can specify the number of bytes to read in the file read method.

file.read([size])

Read at most size bytes from the file (less if the read hits EOF before obtaining size bytes). If the size argument is negative or omitted, read all data until EOF is reached. The bytes are returned as a string object. An empty string is returned when EOF is encountered immediately. (For certain files, like ttys, it makes sense to continue reading after an EOF is hit.) Note that this method may call the underlying C function fread() more than once in an effort to acquire as close to size bytes as possible. Also note that when in non-blocking mode, less data than was requested may be returned, even if no size parameter was given.

You can use seek to move to the next position

file.seek(offset[, whence])

Set the file’s current position, like stdio’s fseek(). The whence argument is optional and defaults to os.SEEK_SET or 0 (absolute file positioning); other values are os.SEEK_CUR or 1 (seek relative to the current position) and os.SEEK_END or 2 (seek relative to the file’s end). There is no return value.

For example, f.seek(2, os.SEEK_CUR) advances the position by two and f.seek(-3, os.SEEK_END) sets the position to the third to last.

Note that if the file is opened for appending (mode 'a' or 'a+'), any seek() operations will be undone at the next write. If the file is only opened for writing in append mode (mode 'a'), this method is essentially a no-op, but it remains useful for files opened in append mode with reading enabled (mode 'a+'). If the file is opened in text mode (without 'b'), only offsets returned by tell() are legal. Use of other offsets causes undefined behavior

Upvotes: 0

Kostas Charitidis
Kostas Charitidis

Reputation: 3103

You can use seek() method which gets as a parameter an offset and moves to the specific byte(character):

offset = 9216  # 9MB
fin.seek(offset, 1)

So you start from 0 and after you've read it you append to offset 9216 or whatever you wish

Upvotes: 1

Related Questions