CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoupcss-selectors
Julian
Julian

Reputation: 152

Scrape nested html with python, search box results

I am trying to understand how to get a list of URLs to scrape from a search box in Cyrillic website. This is the results page, and the search term is "коронавирус" (it's 'windows-1251'): https://www.dnes.bg/search.php?q=%EA%EE%F0%EE%ED%E0%E2%E8%F0%F3%F1

I am trying to get the URLs only under a tag like this one: 

<div class="ttl mb0"><a href="/notifikacii/2020/03/24/greta-tunberg-veroiatno-bila-bolna-ot-covid-19.443414">Грета Тунберг "вероятно" била болна от COVID-19</a></div>

but it's a nested structure. The xpath is:

//*[@id="c1"]/div[4]/div[1]/a

With BeautifulSoup's find_all('a') I find all the links, and I don't need everything, just the search results.

A full-code answer would be most helpful!

Upvotes: 1

Views: 86

Answers (2)

dabingsou
dabingsou

Reputation: 2469

Another solution.

from simplified_scrapy import SimplifiedDoc, req, utils
url = 'https://www.dnes.bg/search.php?q=%EA%EE%F0%EE%ED%E0%E2%E8%F0%F3%F1'
html = '''
<div class="ttl mb0"><a href="/notifikacii/2020/03/24/greta-tunberg-veroiatno-bila-bolna-ot-covid-19.443414">Грета
    Тунберг "вероятно" била болна от COVID-19</a></div>
'''
doc = SimplifiedDoc(html)
urls = doc.selects('div.ttl mb0').a
urls = [(utils.absoluteUrl(url,u.href),u.text) for u in urls]
print (urls)

Result:

[('https://www.dnes.bg/notifikacii/2020/03/24/greta-tunberg-veroiatno-bila-bolna-ot-covid-19.443414', 'Грета Тунберг "вероятно" била болна от COVID-19')]

Upvotes: 1

αԋɱҽԃ αмєяιcαη
αԋɱҽԃ αмєяιcαη

Reputation: 11525

from bs4 import BeautifulSoup
import requests

r = requests.get(
    "https://www.dnes.bg/search.php?q=%EA%EE%F0%EE%ED%E0%E2%E8%F0%F3%F1")
soup = BeautifulSoup(r.content, 'html.parser')


urls = [f"{r.url[:19]}{item.get('href')}" for item in soup.select(
    "div.ttl.mb0 a")]

print(urls)

Output:

['https://www.dnes.bg/notifikacii/2020/03/24/greta-tunberg-veroiatno-bila-bolna-ot-covid-19.443414', 'https://www.dnes.bg/koronavirusat/2020/03/24/kitai-otpuska-merkite-a-evropa-i-sasht-zatiagat-rejima.443411', 'https://www.dnes.bg/mish-mash/2020/03/24/etiket-po-vreme-na-koronavirus-ne-pliuite-i-spazvaite-distanciia.443348', 'https://www.dnes.bg/eu/2020/03/24/jivotyt-v-shveciia-na-pylni-oboroti-koronavirus-li.443384', 'https://www.dnes.bg/akoshtete-vqrvaite/2020/03/24/pri-izolaciia-5-syveta-protiv-preiajdane.443357', 'https://www.dnes.bg/akoshtete-vqrvaite/2020/03/24/po-vreme-na-pandemiia-zashto-panicheski-se-prezapasiavame.443402', 'https://www.dnes.bg/obshtestvo/2020/03/24/v-kriza-podkrepiame-merkite-i-vlastta-strah-ni-e-ot-bezrabotica.443342', 'https://www.dnes.bg/notifikacii/2020/03/24/bolnite-v-italiia-namaliavat-no-bolnicite-vse-oshte-sa-pretovareni.443409', 'https://www.dnes.bg/obshtestvo/2020/03/24/bolnite-ot-koronavirus-u-nas-veche-sa-218.443395', 'https://www.dnes.bg/cars/2020/03/24/avtomobilnite-kompanii-shte-zapochnat-da-proizvejdat-ventilatori.443330', 'https://www.dnes.bg/stranata/2020/03/24/deteto-s-pnevmoniia-v-tyrnovskata-bolnica-bez-vaksini.443405', 'https://www.dnes.bg/koronavirusat/2020/03/24/razrabotiha-inhalator-za-cialostno-lechenie-sreshtu-koronavirus.443295', 'https://www.dnes.bg/koronavirusat/2020/03/24/kiril-domuschiev-prebori-koronavirusa-veche-e-dobre.443401', 'https://www.dnes.bg/koronavirusat/2020/03/24/matematicheski-model-shte-pokazva-licata-pod-karantina-v-burgas.443389', 'https://www.dnes.bg/koronavirusat/2020/03/24/osma-jertva-vze-koronavirusyt-v-rumyniia.443331', 'https://www.dnes.bg/balkani/2020/03/24/syrbiia-nastypva-sreshtu-covid-19-s-masovi-testove.443376', 'https://www.dnes.bg/koronavirusat/2020/03/24/blizo-do-kitai-a-samo-1128-zarazeni-s-koronavirus-kak-go-postigna-iaponiia.443374', 'https://www.dnes.bg/sport/2020/03/24/oficialno-olimpiiskite-igri-shte-se-provedat-prez-2021-g.443380', 'https://www.dnes.bg/sport/2020/03/24/bez-tenis-i-sport-kak-se-podgotvia-viktoriia-tomova-vkyshti.443321', 'https://www.dnes.bg/koronavirusat/2020/03/24/vinovnikyt-za-pandemiiata-ot-covid-19-globalizaciiata.443316']

Please check CSS Selector Reference

Upvotes: 1

Related Questions