Reputation: 25
Trying to scrape the below website.Upon,extracting href from a-tags,it returns none.But,there are elements in those a tags.Here is the code:
from selenium import webdriver
import time
from bs4 import BeautifulSoup
driver = webdriver.Chrome(r"E:\chromedriver_win32\chromedriver.exe")
url= "https://www.adb.org/projects/tenders/sector/information-and-communication-technology-1066"
driver.get(url)
content = driver.page_source.encode('utf-8').strip()
soup = BeautifulSoup(content,"html.parser")
div_tags = soup.findAll("div",{"class":"item-title"})
for tags in div_tags:
a_tag=tags.find('a')
link=a_tag.get('href')
print(link)
div_tags
-output printed:
[<div class="item-title">
<a href="/node/606921"><div class="item-title">Corporate Innovation Expert</div></a></div>, <div class="item-title">Corporate Innovation Expert</div>, <div class="item-title">
<a href="/node/605571"><div class="item-title">Communications Specialist</div></a></div>, <div class="item-title">Communications Specialist</div>, <div class="item-title">
<a href="/node/603231"><div class="item-title">Venture Partner Expert</div></a></div>, <div class="item-title">Venture Partner Expert</div>, <div class="item-title">
<a href="/node/457636"><div class="item-title">Partnerships & Communications Manager</div></a></div>, <div class="item-title">Partnerships & Communications Manager</div>, <div class="item-title">
<a href="/node/545151"><div class="item-title">Operations Associate</div></a></div>, <div class="item-title">Operations Associate</div>, <div class="item-title">
Upvotes: 0
Views: 385
Reputation: 1302
Here is the code:
from requests import get
import time
from bs4 import BeautifulSoup
response = get('https://www.adb.org/projects/tenders/sector/information-and-communication-technology-1066')
soup = BeautifulSoup(response.text, 'html.parser')
div_tags = soup.findAll("div",class_="item-title")
link={}
for tags in div_tags:
a_tag=tags.find('a')
try:
link[a_tag.text]= "https://www.adb.org"+a_tag.get('href')
except:
continue
#print(link)# print all text and value as dict
for x, y in link.items():
print(x, y) # x print text y print link
If you want text and links as a dictionary, use print(link)
.
If you want text, use print(x)
in for loop.
If you want links, use print(y)
in for loop.
Run Online:https://repl.it/repls/ScaredCommonParallelcomputing
Upvotes: 2