How do I deal with list of lists correctly in python? Specifically with a list of functions

Question

I am trying to grep some results pages for work, and then eventually print them out to an html website so someone does not have to manually look through each section.

How I would eventually use: I feed this function a result page, it greps through the 5 different sections, then I can do a html output (thats what that print substitute area is for) with all the different results.

OK MASSIVE EDIT I actually removed the old code because I was asking too many questions. I fixed my code taking some suggestions, but I am still interested in the advantage of using human-readable dict instead of just list. Here is my working code that gets all the right results into a 'list of lists', I then outputted the first section in my eventual html block

import urllib
import re
import string
import sys

def ipv6_results(input_page):
sections = ['/spec.p2/summary.html', '/nd.p2/summary.html',
            '/addr.p2/summary.html', '/pmtu.p2/summary.html', 
            '/icmp.p2/summary.html']
variables_output=[]                                                    
for s in sections:
    temp_list = []
    page = input_page + s
    #print page
    url_reference = urllib.urlopen(page)
    html_page = url_reference.read()
    m = re.search(r'TOTAL</B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
    temp_list.append(int(m.group(1)) )
    m = re.search(r'PASS</B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
    temp_list.append(int(m.group(1)))
    m = re.search(r'FAIL</FONT></B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
    temp_list.append(int(m.group(1)))
    variables_output.append(temp_list)

#print variables to check them :)
print "------"
print variables_output


print "Ready Logo Phase 2"
print "Section                 | Total | Pass | Fail |"
#this next part is eventually going to output an html block
output =  string.Template("""
1 - RFC2460-IPv6 Specs $spec_total $spec_pass $spec_fail
""")
print output.substitute(spec_total=variables_output[0][0], spec_pass=variables_output[0][1],
                        spec_fail=variables_output[0][2])


return 1

imagine the tabbing is correct :( I wish this was more like paste bin, suggestions welcome on pasting code in here

Answer 1

I would use a dictionary inside a list. Maybe something like:

def ipv6_results(input_page):
    results = [{file_name:'/spec.p2/summary.html', total:0, pass:0, fail:0},
               {file_name:'/nd.p2/summary.html', total:0, pass:0, fail:0},       
               {file_name:'/addr.p2/summary.html', total:0, pass:0, fail:0},
               {file_name:'/pmtu.p2/summary.html', total:0, pass:0, fail:0},
               {file_name:'/icmp.p2/summary.html', total:0, pass:0, fail:0}] 
    for r in results:
        url_reference = urllib.urlopen(input_page + r[file_name])
        html_page = url_reference.read()
        m = re.search(r'TOTAL</B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
        r[total] = int(m.group(1))
        m = re.search(r'PASS</B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
        r[pass] = int(m.group(1))
        m = re.search(r'FAIL</FONT></B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
        r[fail] = int(m.group(1))

    for r in results:
        print r[total]
        print r[pass]
        print r[fail]

Answer 2

Generally, you don't declare the shape of the list first, and then fill in the values. Instead, you build the list as you discover the values.

Your variables has a lot of structure. You've got inner lists of 3 elements, always in the order of 'total', 'pass', 'fail'. Perhaps these 3-tuples should be made namedtuples. That way, you can access the three parts with humanly-recogizable names (data.total, data.pass, data.fail), instead of cryptic index numbers (data[0], data[1], data[2]).

Next, your 3-tuples differ by prefixes: 'spec', 'nd', 'addr', etc. These sound like keys to a dict rather than elements of a list.

So perhaps consider making variables a dict. That way, you can access the particular 3-tuple you want with the humanly-recognizable variables['nd'] instead of variables[1]. And you can access the nd_fail value with variables['nd'].fail instead of variables[1][2]:

import collections
# define the namedtuple class Point (used below).
Point = collections.namedtuple('Point', 'total pass fail')

# Notice we declare `variables` empty at first; we'll fill in the values later.
variables={}
keys=('spec','nd','addr','pmtu','icmp')
for s in sections:
    for key in keys:
        page = input_page + s
        url_reference = urllib.urlopen(page)
        html_page = url_reference.read()
        m = re.search(r'TOTAL</B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
        ntotal = int(m.group(1)) 
        m = re.search(r'PASS</B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
        npass = int(m.group(1))
        m = re.search(r'FAIL</FONT></B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
        nfail = int(m.group(1))

        # We create an instance of the namedtuple on the right-hand side
        # and store the value in `variables[key]`, thus building the 
        # variables dict incrementally.
        variables[key]=Point(ntotal,npass,nfail)

Answer 3

The first thing is that those lists there will only be the values of the variables, at the time of assignment. You would be changing the list value, but not the variables.

I would seriously consider using classes and build structures of those, including lists of class instances.

For example:

class SectionResult:
  def __init__(self, total = 0, pass = 0, fail = 0):
    self.total = total
    self.pass = pass
    self.fail = fail

Since it looks like each group should link up with a section, you can create a list of dictionaries (or perhaps a list of classes?) with the bits associated with a section:

sections = [{'results' : SectionResult(), 'filename': '/addr.p2/summary.html'}, ....]

Then in the loop:

for section in sections:
   page = input_page + section['filename']
   url_reference = urllib.urlopen(page)
   html_page = url_reference.read()
   m = re.search(r'TOTAL</B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
   section['results'].total = int(m.group(1)) 
   m = re.search(r'PASS</B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
   section['results'].pass = int(m.group(1))
   m = re.search(r'FAIL</FONT></B></TD><TD>:</TD><TD>([0-9,]+)', html_page)
   section['results'].fail = int(m.group(1))