CODEWITHSUNDEEP
pythonregexbeautifulsoup
Orest Tokovenko
Orest Tokovenko

Reputation: 173

How can I filter for a keyword in a list of strings?

I have a list of strings which are links that I scraped using BeautifulSoup. I cannot figure out how to return only the strings which contain the word 'The'. The solution might use Regex but it has not worked for me.

I tried

for i in links_list:
     if re.match('^The', i) is not None:
        eps_only.append(i)

But I get errors like

File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.8/re.py", line 191, in match
    return _compile(pattern, flags).match(string)
TypeError: expected string or bytes-like object

The list looks like this:

['index.html', 'seinfeld-scripts.html', 'episodes_oveview.html', 'seinfeld-characters.html', 'buy-seinfeld.html', 'http://addthis.com/bookmark.php?v=250&username=doctoroids', None, None, None, None, 'http://community.seinfeldscripts.com', 'buy-seinfeld.html', 'seinfeld-t-shirt.html', 'seinfeld-dvd.html', 'episodes_oveview.html', 'alpha.html', '    http://www.shareasale.com/r.cfm?u=439896&b=119192&m=16934&afftrack=seinfeldScriptsTop&urllink=search%2E80stees%2Ecom%2F%3Fcategory%3D80s%2BTV%26i%3D1%26theme%3DSeinfeld%26u1%3Dcategory%26u2%3Dtheme', ' TheSeinfeldChronicles.htm', ' TheStakeout.htm', ' TheRobbery.htm', ' MaleUnbonding.htm', ' TheStockTip.htm', ' TheExGirlfriend.htm', ' ThePonyRemark.htm', ' TheJacket.htm', ' ThePhoneMessage.htm', ' TheApartment.htm', ' TheStatue.htm', ' TheRevenge.htm', ' TheHeartAttack.htm', ' TheDeal.htm', ' TheBabyShower.htm', ' TheChineseRestaurant.htm', ' TheBusboy.htm', 'TheNote.html', ' TheTruth.htm', 'ThePen.html', ' TheDog.htm', ' TheLibrary.htm', ' TheParkingGarage.htm', 'TheCafe.html', ' TheTape.htm', 'TheNoseJob.html', 'TheStranded.html', ...]

Update: Full Code

import requests
import re
from bs4 import BeautifulSoup

##################
##--user agent--##
##################

user_agent_desktop = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) '\
    'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 '\
    'Safari/537.36'

headers = {'User-Agent': user_agent_desktop}

#########################
##--fetching the page--##
#########################

URL = 'https://www.seinfeldscripts.com/seinfeld-scripts.html'
page = requests.get(URL, headers=headers)
soup = BeautifulSoup(page.content, 'html.parser')


############################################################
##--scraping the links to the scripts from the main page--##
############################################################

links_list = []
eps_only = []

for link in soup.find_all('a'):
    links_list.append(link.get('href'))

### sorting for links that contain 'the' ###

for i in filter(None, links_list):
    if re.match('^The', str(i)) is not None:
        eps_only.append(i)
        print(eps_only)

Upvotes: 1

Views: 74

Answers (2)

Jarvis
Jarvis

Reputation: 8564

You should filter the list elements (without None) returned from BeautifulSoup:

for i in filter(None, links_list):
    if re.match('^The', str(i)) is not None:
        eps_only.append(i)
print(eps_only)

Upvotes: 1

costaparas
costaparas

Reputation: 5237

Python's re.match will fail if it gets passed None as an argument -- hence the error you're getting.

Some of your list elements are None.

You will have to check for such elements before passing them to re.match.

For example:

for i in links_list:
    if i is not None and re.match('^The', i) is not None:
        eps_only.append(i)

Or, you could filter them out prior, like this:

links_list = [l for l in links_list if l is not None]

Upvotes: 1

Related Questions