CODEWITHSUNDEEP
pythonlistweb-scrapingbeautifulsoup
Alex
Alex

Reputation: 655

Scrape all href into list with BeautifulSoup

I'd like to to grab links from this page and put them in a list.

I have this code:

import bs4 as bs
import urllib.request

source = urllib.request.urlopen('http://www.gcoins.net/en/catalog/236').read()
soup = bs.BeautifulSoup(source,'lxml')

links = soup.find_all('a', attrs={'class': 'view'})
print(links)

It produces following output:

[<a class="view" href="/en/catalog/view/514">
<img alt="View details" height="32" src="/img/actions/file.png" title="View details" width="32"/>
</a>, 

     """There are 28 lines more"""

      <a class="view" href="/en/catalog/view/565">
<img alt="View details" height="32" src="/img/actions/file.png" title="View details" width="32"/>
</a>]

I need to get following: [/en/catalog/view/514, ... , '/en/catalog/view/565']

But then I go ahead and add following: href_value = links.get('href') I got an error.

Upvotes: 1

Views: 386

Answers (3)

user15763208
user15763208

Reputation: 1

Try the code below. You get the HTML list in one step:

import bs4 as bs
import urllib.request

source = urllib.request.urlopen('http://www.gcoins.net/en/catalog/236').read()
soup = bs.BeautifulSoup(source,'lxml')

links = [i.get("href") for i in soup.find_all('a', attrs={'class': 'view'})]
for link in links:
    print('http://www.gcoins.net'+ link)

Upvotes: 0

ExtractTable.com
ExtractTable.com

Reputation: 811

Your links is currently a python list. What you want to do is loop into that list and fetch the hrefs as below.

final_hrefs = []
for each_link in links:
    final_hrefs.append(each_link.a['href'])

or a one-liner

final_hrefs = [each_link['href'] for each_link in links]

print(final_hrefs)

Upvotes: 1

Rakesh
Rakesh

Reputation: 82765

Try:

soup = bs.BeautifulSoup(source,'lxml')

links = [i.get("href") for i in soup.find_all('a', attrs={'class': 'view'})]
print(links)

Output:

['/en/catalog/view/514', '/en/catalog/view/515', '/en/catalog/view/179080', '/en/catalog/view/45518', '/en/catalog/view/521', '/en/catalog/view/111429', '/en/catalog/view/522', '/en/catalog/view/182223', '/en/catalog/view/168153', '/en/catalog/view/523', '/en/catalog/view/524', '/en/catalog/view/60228', '/en/catalog/view/525', '/en/catalog/view/539', '/en/catalog/view/540', '/en/catalog/view/31642', '/en/catalog/view/553', '/en/catalog/view/558', '/en/catalog/view/559', '/en/catalog/view/77672', '/en/catalog/view/560', '/en/catalog/view/55377', '/en/catalog/view/55379', '/en/catalog/view/32001', '/en/catalog/view/561', '/en/catalog/view/562', '/en/catalog/view/72185', '/en/catalog/view/563', '/en/catalog/view/564', '/en/catalog/view/565']

Upvotes: 1

Related Questions