CODEWITHSUNDEEP
pythonbeautifulsoup
Aakash Hirve
Aakash Hirve

Reputation: 77

BeautifulSoup : Fetched all the links on a webpage how to navigate through them without selenium?

So I'm trying to write a mediocre script to download subtitles from one particular website as y'all can see. I'm a newbie to beautifulsoup, so far I have a list of all the "href" after a search query(GET). So how do I navigate further, after getting all the links? Here's the code:

import requests
from bs4 import BeautifulSoup

usearch = input("Movie Name? : ")
url = "https://www.yifysubtitles.com/search?q="+usearch
print(url)
resp = requests.get(url)
soup = BeautifulSoup(resp.content, 'lxml')
for link in soup.find_all('a'):
    dictn = link.get('href')
    print(dictn)

Upvotes: 1

Views: 60

Answers (1)

Rahul
Rahul

Reputation: 11560

You need to use resp.text instead of resp.content Try this to get the search results.

import requests
from bs4 import BeautifulSoup

base_url_f = "https://www.yifysubtitles.com"
search_url = base_url_f + "/search?q=last+jedi"
resp = requests.get(search_url)
soup = BeautifulSoup(resp.text, 'lxml')
for media in soup.find_all("div", {"class": "media-body"}):
    print(base_url_f + media.find('a')['href'])

out: https://www.yifysubtitles.com/movie-imdb/tt2527336

Upvotes: 1

Related Questions