Loops in python does not loop through every line in file

Question

I am trying to loop through every line in a text file and perform some actions. Right now I have a text file which contains this:

--- small modified --- #line 1
1,2,3                  #line 2
4,5,6                  #line 3
--- big modified ---   #line 4
7;8;9                  #line 5
10;11;12               #line 6

I am trying to parse line 2,3 into one file, and lines 5,6 into another file but right now, only lines 2 and 3 gets written into the file and idk why the "elif" statement is not run. I can't solve the logic error and would appreciate if someone could help me out.

Below is my code:

def convert_json(fileName):
    with open(fileName,'r') as file:
        for line in file:
            if 'modified' and 'small' in line:
                for li in file:
                    fields1 = li.split(',')
                    if len(fields1) >= 3:
                            smallarr.append({
                            "a": fields1[0],
                            "b": fields1[1],
                            "c": fields1[2]
                                })
                            with open('smalljson.txt','w+') as small_file:
                                json.dump(smallarr, small_file)
                    else:
                        pass

            elif 'modified' and 'big' in line:
                for li in file:
                    fields2 = li.split(';')
                    if len(fields2) >= 3:
                            bigarr.append({
                            "w1": fields2[0],
                            "w2": fields2[1],
                            "w3": fields2[2],
                                })
                            with open('big.txt','w+') as big_file:
                                json.dump(bigarr, big_file)
                    else: 
                        pass



            else:
                print 'test'

Update: THis is my current code, I am able to do it but only for lines 2 and lines 5, other than s second for-loop i cannot think of another way to loop through the lines

def convert_json(fileName):
with open(fileName,'r') as file:
    for line in file:
        #if 'modified' in line and 'small' in line:
        if 'modified' in line and 'Small' in line:
            fields1 = next(file).split(',')
            if len(fields1) >= 3:
                smallarr.append({
                "a": fields1[0],
                "b": fields1[1],
                "c": fields1[2]
                })
                with open('smalljson.txt','w+') as small_file:
                    json.dump(smallarr, small_file)
            else:
                pass



        elif 'modified' in line and 'big' in line:
            fields2 = next(file).split(';')
            if len(fields2) >= 3:
                bigarr.append({
                "w1": fields2[0],
                "w2": fields2[1],
                "w3": fields2[2],
                })
                with open('bigwater.txt','w+') as big_file:
                    json.dump(bigarr, big_file)
            else:
                pass

        else:
            print 'test'

Answer 1

There are a few issues in your code.

Firstly you are repeating yourself. The big and small cases don't vary enough to justify the code duplication.

Secondly, while I understand what you're trying to do with next(file), you'd still need to loop that instruction in some way to go get the next lines. But wait, you're already doing that exactly with for line in file.

Finally, at each loop, you're reopening the same file and redumping an ever increasing array. This is wasteful IO. If you're trying to stream from file into bigwater.txt and smalljson.txt and not storing too much stuff in memory, this is the wrong approach since json.dump can't be used to stream data.

Here's my take at it:

def convert_json(fileName):
    big = []
    small = []
    with open(fileName,'r') as file:
        for line in file:
            line = line.strip()
            if line.startswith("--"):
                if "big" in line:
                    array = big
                    keys = ["w1", "w2", "w3"]
                    sep = ";"
                else:
                    array = small
                    keys = ["a", "b", "c"]
                    sep = ","
                continue

            values = line.split(sep)
            # todo: make sure sizes match
            mapping = dict(zip(keys, values))
            array.append(mapping)

    with open('smalljson.txt','w') as small_file:
        json.dump(small, small_file)
    with open('bigwater.txt','w') as big_file:
        json.dump(big, big_file)

Answer 2

Your parsing logic need to be changed. Here is what code looks like, use it for reference in future improvements.

def file_parser(self):
    file_section = 0

    smallarr = []
    bigarr = []
    with open('data.txt') as in_file:
        for in_line in in_file:
            in_line = in_line.strip()

            if 'small' in in_line:
                file_section = 1
                continue
            elif 'big' in in_line:
                file_section = 2
                continue

            if file_section == 1:
                fields1 = in_line.split(',')
                if len(fields1) >= 3:
                    smallarr.append({
                        "a": fields1[0],
                        "b": fields1[1],
                        "c": fields1[2]
                    })
            elif file_section == 2:
                fields2 = in_line.split(';')
                if len(fields2) >= 3:
                    bigarr.append({
                        "w1": fields2[0],
                        "w2": fields2[1],
                        "w3": fields2[2],
                    })

    with open('small.txt', 'w+') as small_file:
        json.dump(smallarr, small_file)

    with open('big.txt', 'w+') as big_file:
        json.dump(bigarr, big_file)

Input data:

--- small modified ---
1,2,3
4,5,6
--- big modified ---
7;8;9
10;11;12

small.txt

[{"a": "1", "c": "3", "b": "2"}, {"a": "4", "c": "6", "b": "5"}]

big.txt

[{"w3": "9", "w2": "8", "w1": "7"}, {"w3": "12", "w2": "11", "w1": "10"}]

Answer 3

change

elif 'modified' and 'big' in line:

into

elif 'modified' in line and 'big' in line: