CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Sean Kelly
Sean Kelly

Reputation: 183

Short & Easy - soup.find_all Not Returning Multiple Tag Elements

I need to scrape all 'a' tags with "result-title" class, and all 'span' tags with either class 'results-price' and 'results-hood'. Then, write the output to a .csv file across multiple columns. The current code does not print anything to the csv file. This may be bad syntax but I really can't see what I am missing. Thanks.

f = csv.writer(open(r"C:\Users\Sean\Desktop\Portfolio\Python - Web Scraper\RE Competitor Analysis.csv", "wb"))

def scrape_links(start_url):
for i in range(0, 2500, 120):
    source = urllib.request.urlopen(start_url.format(i)).read()
    soup = BeautifulSoup(source, 'lxml')
    for a in soup.find_all("a", "span", {"class" : ["result-title hdrlnk", "result-price", "result-hood"]}):
        f.writerow([a['href']], span['results-title hdrlnk'].getText(), span['results-price'].getText(), span['results-hood'].getText() )
    if i < 2500:
        sleep(randint(30,120))
    print(i)


scrape_links('my_url')

Upvotes: 1

Views: 75

Answers (1)

Imran
Imran

Reputation: 13458

If you want to find multiple tags with one call to find_all, you should pass them in a list. For example:

soup.find_all(["a", "span"])

Without access to the page you are scraping, it's too hard to give you a complete solution, but I recommend extracting one variable at a time and printing it to help you debug. For example:

a = soup.find('a', class_ = 'result-title')
a_link = a['href']
a_text = a.text

spans = soup.find_all('span', class_ = ['results-price', 'result-hood'])

row = [a_link, a_text] + [s.text for s in spans]
print(row) # verify we are getting the results we expect

f.writerow(row)

Upvotes: 1

Related Questions