Counting words in a dictionary (Python)

Question

I have this code, which I want to open a specified file, and then every time there is a while loop it will count it, finally outputting the total number of while loops in a specific file. I decided to convert the input file to a dictionary, and then create a for loop that every time the word while followed by a space was seen it would add a +1 count to WHILE_ before finally printing WHILE_ at the end.

However this did not seem to work, and I am at a loss as to why. Any help fixing this would be much appreciated.

This is the code I have at the moment:

WHILE_ = 0
INPUT_ = input("Enter file or directory: ")


OPEN_ = open(INPUT_)
READLINES_ = OPEN_.readlines()
STRING_ = (str(READLINES_))
STRIP_ = STRING_.strip()
input_str1 = STRIP_.lower()


dic = dict()
for w in input_str1.split():
    if w in dic.keys():
        dic[w] = dic[w]+1
    else:
        dic[w] = 1
DICT_ = (dic)


for LINE_ in DICT_:
    if  ("while\\n',") in LINE_:
        WHILE_ += 1
    elif ('while\\n",') in LINE_:
        WHILE_ += 1
    elif ('while ') in LINE_:
        WHILE_ += 1

print ("while_loops {0:>12}".format((WHILE_)))

This is the input file I was working from:

'''A trivial test of metrics
Author: Angus McGurkinshaw
Date: May 7 2013
'''

def silly_function(blah):
    '''A silly docstring for a silly function'''
    def nested():
        pass
    print('Hello world', blah + 36 * 14)
    tot = 0  # This isn't a for statement
    for i in range(10):
        tot = tot + i
        if_im_done = false  # Nor is this an if
    print(tot)

blah = 3
while blah > 0:
    silly_function(blah)
    blah -= 1
    while True:
        if blah < 1000:
            break

The output should be 2, but my code at the moment prints 0

Answer 1

This is an incredibly bizarre design. You're calling readlines to get a list of strings, then calling str on that list, which will join the whole thing up into one big string with the quoted repr of each line joined by commas and surrounded by square brackets, then splitting the result on spaces. I have no idea why you'd ever do such a thing.

Your bizarre variable names, extra useless lines of code like DICT_ = (dic), etc. only serve to obfuscate things further.

But I can explain why it doesn't work. Try printing out DICT_ after you do all that silliness, and you'll see that the only keys that include while are while and 'while. Since neither of these match any of the patterns you're looking for, your count ends up as 0.

It's also worth noting that you only add 1 to WHILE_ even if there are multiple instances of the pattern, so your whole dict of counts is useless.

---

This will be a lot easier if you don't obfuscate your strings, try to recover them, and then try to match the incorrectly-recovered versions. Just do it directly.

While I'm at it, I'm also going to fix some other problems so that your code is readable, and simpler, and doesn't leak files, and so on. Here's a complete implementation of the logic you were trying to hack up by hand:

import collections

filename = input("Enter file: ")
counts = collections.Counter()
with open(filename) as f:
    for line in f:
        counts.update(line.strip().lower().split())
print('while_loops {0:>12}'.format(counts['while']))

When you run this on your sample input, you correctly get 2. And extending it to handle if and for is trivial and obvious.

---

However, note that there's a serious problem in your logic: Anything that looks like a keyword but is in the middle of a comment or string will still get picked up. Without writing some kind of code to strip out comments and strings, there's no way around that. Which means you're going to overcount if and for by 1. The obvious way of stripping—line.partition('#')[0] and similarly for quotes—won't work. First, it's perfectly valid to have a string before an if keyword, as in "foo" if x else "bar". Second, you can't handle multiline strings this way.

These problems, and others like them, are why you almost certainly want a real parser. If you're just trying to parse Python code, the ast module in the standard library is the obvious way to do this. If you want to be write quick&dirty parsers for a variety of different languages, try pyparsing, which is very nice, and comes with some great examples.

Here's a simple example:

import ast

filename = input("Enter file: ")
with open(filename) as f:
    tree = ast.parse(f.read())
while_loops = sum(1 for node in ast.walk(tree) if isinstance(node, ast.While))
print('while_loops {0:>12}'.format(while_loops))

Or, more flexibly:

import ast
import collections

filename = input("Enter file: ")
with open(filename) as f:
    tree = ast.parse(f.read())
counts = collections.Counter(type(node).__name__ for node in ast.walk(tree))    
print('while_loops {0:>12}'.format(counts['While']))
print('for_loops {0:>14}'.format(counts['For']))
print('if_statements {0:>10}'.format(counts['If']))