CODEWITHSUNDEEP
pythonweb-scraping
Rachel9866
Rachel9866

Reputation: 131

Searching htmls by text. Error: string indices must be integers

I am trying to webscrape to some pdfs on a local council website. I only want certain dates though, is it possible to search them by text?

So for example, I want the ones from certain months.

I've written my code to find these but it's giving me this error:

TypeError: string indices must be integers

And it's for the line of text which has the dates in.

Here's my code: 

import requests
import urllib.request
import time 
from bs4 import BeautifulSoup as bs

url = "https://www.gmcameetings.co.uk"

meeting_links = soup.find('a', {'href':"https://www.gmcameetings.co.uk/meetings/committee/36/economy_business_growth_and_skills_overview_and_scrutiny"})

f = open(r"E:\Internship\WORK\GMCA\Getting PDFS\gmcabusinessminutelinks.txt", "w+")

for link in meeting_links:
    if link['text'].find_all(["April 2018"],["May 2018"],["June 2018"],["July 2018"])>1:
        r2 = requests.get(link['href'])
        print("link1")
        page2 = r2.text
        soup2 = bs(page2, 'lxml')
        pdf_links = soup2.find_all('a', href=True)
        for plink in pdf_links:
            if plink['href'].find('minutes')>1:
                print("Minutes!")
                f.write(str(plink['href']) + ' ')
f.close()

Is it possible to do this, or is it the way I've written it?

Upvotes: 0

Views: 42

Answers (1)

QHarr
QHarr

Reputation: 84455

You can use :contains with bs4 4.7.1.

import requests
from bs4 import BeautifulSoup as bs

dates = ['July 2019', 'December 2018']
r = requests.get('https://www.gmcameetings.co.uk/meetings/committee/36/economy_business_growth_and_skills_overview_and_scrutiny')
soup = bs(r.content, 'lxml')

links = []

for date in dates:
    l = [item['href'] for item in soup.select('a:contains("' + date + '")')]
    links.append(l)

Flatten the list at the end:

final = [i for item in links for i in item]

Upvotes: 1

Related Questions