Comparing to a list in python using BeautifulSoup

Question

The following code is meant to look through tags in a webpage (the 'b', 'strong' and 'a' tags in specific 'li' entries). If the tag is found in a list (which can be found in the code) then the 'a class=vote-description__evidence' tag is added to another list - otherwise 0 is added to this list. The code can be found here:

import urllib2
from BeautifulSoup import *

def votedescget(link):
    response = urllib2.urlopen(link)
    html = response.read()
    soup = BeautifulSoup(html)
    desc = soup.findAll('ul',{'class':"vote-descriptions"})
    readVotes = open("categories.txt","r")
    #descList = []

    #for line in readVotes.read().splitlines():
        #descList.append(line)

    resultsList = []
    descList = ['gay rights', 'smoking bans', 'hunting ban', 'marriage', 'equality and human rights', 'assistance to end their life', 'UK military forces', 'Iraq war', 'investigations', 'Trident', 'EU integration', 'EU', 'Military Covenant', 'right to remain for EU nationals', 'UK membership of the EU', 'military action against ISIL (Daesh)', 'housing benefit', 'welfare benefits', 'illness or disability', 'council tax', 'welfare benefits', 'guaranteed jobs for young people', 'income tax', 'rate of VAT', 'alcoholic drinks', 'taxes on plane tickets', 'fuel for motor vehicles', 'income over £150,000', 'occupational pensions', 'occupational pensions', 'banker’s bonus tax', 'taxes on banks', 'mansion tax', 'rights for shares', 'regulation of trade union activity', 'capital gains tax', 'corporation tax', 'tax avoidance', 'incentives for companies to invest', 'high speed rail', 'private patients', 'NHS', 'foundation hospitals', 'smoking bans', 'assistance to end their life', 'autonomy for schools', 'undergraduate tuition fee', 'academy schools', 'financial support', 'tuition fees', 'funding of local government', 'equal number of electors', 'fewer MPs', 'transparent Parliament', 'proportional system', 'wholly elected', 'taxes on business premises', 'campaigning by third parties', 'fixed periods between parliamentary elections', 'hereditary peers', 'more powers to the Welsh Assembly', 'more powers to the Scottish Parliament', 'powers for local councils', 'over laws specifically impacting their part of the UK', 'voting age', 'stricter asylum system', 'intervene in inquests', 'ID cards', 'Police and Crime Commissioners', 'retention of information about communications', 'enforcement of immigration rules', 'mass surveillance', 'merging police and fire services', 'prevent climate change', 'fuel for motor vehicles', 'forests', 'taxes on plane tickets', 'electricity generation', 'culling badgers', 'hydraulic fracturing (fracking)', 'high speed rail', 'bus services', 'rail fares', 'fuel for motor vehicles', 'taxes on plane tickets', 'publicly owned railway system', 'secure tenancies for life', 'market rent to high earners renting a council home', 'regulation of gambling', 'civil service redundancy payments', 'anti-terrorism laws', 'Royal Mail', 'pub landlords rent-only leases', 'legal aid', 'courts in secret sessions', 'register of lobbyists', 'no-win no fee cases', 'letting agents', 'Conservative - Liberal Democrat Coalition Agreement']
    #print descList

    for line in desc:
        li_list = line.findAll('li')
        for li in li_list:
            if len(li.findAll('b')) == 1:
                if li.find('b') in descList:
                    resultsList.append(str(li.find('a',{'class':"vote-description__evidence"})))
                    print li.find('a',{'class':"vote-description__evidence"})
            elif len(li.findAll('b')) == 2:
                print li.findAll('b')[1]
                if li.findAll('b')[1] in descList:
                    resultsList.append(str(li.find('a',{"class':'vote-description__evidence"})))
                    print li.find('a',{'class':"vote-description__evidence"})
            elif li.find('strong') in descList:
                resultsList.append(str(li.find('a',{"class':'vote-description__evidence"})))
                print li.find('a',{'class':"vote-description__evidence"})
            elif li.find('a') in descList:
                resultsList.append(str(li.find('a',{"class':'vote-description__evidence"})))
                print li.find('a',{'class':"vote-description__evidence"})
            else:
                resultsList.append('0')

  print resultsList

votedescget("https://www.theyworkforyou.com/mp/10001/diane_abbott/hackney_north_and_stoke_newington/votes")

Usually the list is created programmatically from a file but for the sake of ease I've just included it as a variable. For some reason the result I'm getting when I run this code is as follows:

assistance to end their life
council tax
assistance to end their life
over laws specifically impacting their part of the UK
electricity generation
no-win no fee cases
letting agents
['0', '0', '0', '0']

Could anyone tell me why this is happening, or how to fix it? What I'm expecting is a list of zeroes interspersed with results where the tags are found in descList, but this isn't what's happening.

B.Adler · Accepted Answer

In your comparison you are checking if li.find('b') in descList: Have you tested whether or not a navigable string can be compared to a string in this way? Beautiful soup returns a navigable string rather than a string, which is why you are type casting it to a string before you append it to your list; however, you are not type casting it before this comparison.

Comparing to a list in python using BeautifulSoup

Answers (1)

Related Questions