CODEWITHSUNDEEP
python
Teeps
Teeps

Reputation: 41

Python: Read large file in chunks

Hey there, I have a rather large file that I want to process using Python and I'm kind of stuck as to how to do it.

The format of my file is like this:

0 xxx xxxx xxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1 xxx xxxx xxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

So I basically want to read in the chunk up from 0-1, do my processing on it, then move on to the chunk between 1 and 2.

So far I've tried using a regex to match the number and then keep iterating, but I'm sure there has to be a better way of going about this. Any suggestion/info would be greatly appreciated.

Upvotes: 4

Views: 9786

Answers (6)

eyquem
eyquem

Reputation: 27575

If the file's content can be loaded in memory, and that's what you answered, then the following code (needs to have filename defined) may be a solution.

import re

regx = re.compile('^((\d+).*?)(?=^\d|\Z)',re.DOTALL|re.MULTILINE)

with open(filename) as f:
    text = f.read()

def treat(inp,regx=regx):
    m1  = regx.search(inp)
    numb,chunk = m1.group(2,1)
    li = [chunk]
    for mat in regx.finditer(inp,m1.end()):
        n,ch = mat.group(2,1)
        if int(n) == int(numb) + 1:
            yield ''.join(li)
            numb = n
            li = []
        li.append(ch)
        chunk = ch
    yield ''.join(li)

for y in treat(text):
    print repr(y)

This code, run on a file containing :

1 mountain
orange 2
apple
produce
2 gas
solemn
enlightment
protectorate
3 grimace
song
4 snow
wheat
51 guludururu
kelemekinonoto
52asabi dabada
5 yellow
6 pink 
music
air
7 guitar
blank 8
8 Canada
9 Rimini

produces:

'1 mountain\norange 2\napple\nproduce\n'
'2 gas\nsolemn\nenlightment\nprotectorate\n'
'3 grimace\nsong\n'
'4 snow\nwheat\n51 guludururu\nkelemekinonoto\n52asabi dabada\n'
'5 yellow\n'
'6 pink \nmusic\nair\n'
'7 guitar\nblank 8\n'
'8 Canada\n'
'9 Rimini'

Upvotes: 0

user166390
user166390

Reputation:

If the "N " can only start a line, then why not use use the "simple" solution? (It sounds like this already being done, I am trying to reinforce/support it ;-))

That is, just reading a line at a time, and build up the data representing the current N object. After say N=0, and N=1 are loaded, process them together, then move onto the next pair (N=2, N=3). The only thing that is even remotely tricky is making sure not to throw out a read line. (The line read that determined the end condition -- e.g. "N " -- also contain the data for the next N).

Unless seeking is required (or IO caching is disabled or there is an absurd amount of data per item), there is really no reason not to use readline AFAIK.

Happy coding.

Here is some off-the-cuff code, which likely contains multiple errors. In any case, it shows the general idea using a minimized side-effect approach.

# given an input and previous item data, return either
# [item_number, data, next_overflow] if another item is read
# or None if there are no more items
def read_item (inp, overflow):
  data = overflow or ""

  # this can be replaced with any method to "read the header"
  # the regex is just "the easiest". the contract is just:
  # given "N ....", return N. given anything else, return None
  def get_num(d):
    m = re.match(r"(\d+) ", d)
    return int(m.groups(1)) if m else None

  for line in inp:
    if data and get_num(line) ne None:
      # already in an item (have data); current line "overflows".
      # item number is still at start of current data
      return [get_num(data), data, line]

    # not in item, or new item not found yet
    data += line

  # and end of input, with data. only returns above
  # if a "new" item was encountered; this covers case of
  # no more items (or no items at all)
  if data:
    return [get_num(data), data, None]
  else
    return None

And usage might be akin to the following, where f represents an open file:

# check for error conditions (e.g. None returned)
# note feed-through of "overflow"
num1, data1, overflow = read_item(f, None)
num2, data2, overflow = read_item(f, overflow)

Upvotes: 1

wheaties
wheaties

Reputation: 35970

If they are all within the same line, that is there are no line breaks between "1." and "2." then you can iterate over the lines of the file like this:

for line in open("myfile.txt"):
    #do stuff

The line will be disposed of and overwritten at each iteration meaning you can handle large file sizes with ease. If they're not on the same line:

for line in open("myfile.txt"):
    if #regex to match start of new string
       parsed_line = line
    else:
       parsed_line += line

and the rest of your code.

Upvotes: 3

phimuemue
phimuemue

Reputation: 35983

Why don't you just read the file char by char using file.read(1)?

Then, you could - in each iteration - check whether you arrived at the char 1. Then you have to make sure that storing the string is fast.

Upvotes: 2

aterimperator
aterimperator

Reputation: 199

If the file is small, you could read the whole file in and split() on number digits (might want to use strip() to get rid of whitespace and newlines), then fold over the list to process each string in the list. You'll probably have to check that the resultant string you are processing on is not initially empty in case two digits were next to each other.

Upvotes: 0

tMC
tMC

Reputation: 19315

If the format is fixed, why not just read 3 lines at a time with readline()

Upvotes: 0

Related Questions