CODEWITHSUNDEEP
pythonxmlweb-scrapingbeautifulsoup
ColeWorld
ColeWorld

Reputation: 279

Scraping for href links

Trying to collect the specific link on this page with the correct keywords, so far I have: 

from bs4 import BeautifulSoup
import random
url = 'http://www.thenextdoor.fr/en/4_adidas-originals'
r = requests.get(url)
soup = BeautifulSoup(r.text, 'lxml')
raw = soup.findAll('a', {'class':'add_to_compare'})
links = raw['href']
keyword1 = 'adidas'
keyword2 = 'thenextdoor'
keyword3 = 'uncaged'
for link in links:
    text = link.text
    if keyword1 in text and keyword2 in text and keyword3 in text:

Im trying to extract this link

Upvotes: 3

Views: 1342

Answers (2)

alecxe
alecxe

Reputation: 474003

Or, you can do it in one go using a function as an href attribute value for find_all():

keywords = ['adidas', 'thenextdoor', 'Uncaged']
links = soup.find_all('a',
                      class_='add_to_compare',
                      href=lambda href: all(keyword in href for keyword in keywords))
for link in links:  
    print(link["href"])

Upvotes: 1

Mohammad Yusuf
Mohammad Yusuf

Reputation: 17074

You can check if all are present with all() and if either 1 is present with any()

from bs4 import BeautifulSoup
import requests

res = requests.get("http://www.thenextdoor.fr/en/4_adidas-originals").content
soup = BeautifulSoup(res)

atags = soup.find_all('a', {'class':'add_to_compare'})
links = [atag['href'] for atag in atags]
keywords = ['adidas', 'thenextdoor', 'Uncaged']

for link in links:  
    if all(keyword in link for keyword in keywords):
        print link

Output:

http://www.thenextdoor.fr/en/clothing/2042-adidas-originals-Ultraboost-Uncaged-2303002052017.html
http://www.thenextdoor.fr/en/clothing/2042-adidas-originals-Ultraboost-Uncaged-2303002052017.html

Upvotes: 3

Related Questions