CODEWITHSUNDEEP
pythonbeautifulsoup
growntreee
growntreee

Reputation: 21

Adding text to results in an array using beautifulsoup in python

Anyone any ideas how to prepend each item in an array with text before its passed into the next loop?

Basically I have found the links that im after but they do not contain the main sites url, just the child elements

links = []
for link in soup.find_all("a", {"class": "product-info__caption"}):
links.append(link.attrs['href'])

#this returns the urls okay as /products/item
#whereas i need the https://www.example.com/products/item to pass into next loop

for x in links:

result = requests.get(x)
src = result.content
soup = BeautifulSoup(src, 'lxml')
Name = soup.find('h1', class_='product_name')
... and so on

Upvotes: 1

Views: 112

Answers (2)

theX
theX

Reputation: 1134

Building on top of @Andrej Kesely's answer, I think you should use a list comprehension

links = [
   "https://www.example.com" + link.attrs['href']
   for link in soup.find_all("a", {"class": "product-info__caption"})
]

List comprehensions are faster than the conventional for loop. This StackOverflow answer will explain why list comprehensions are faster.

Note

Every list comprehension can be turned into a for loop.

Further reading

Real Python has an amazing article about them here.

Official Python documentation about list comprehensions can be found here.

Upvotes: 0

Andrej Kesely
Andrej Kesely

Reputation: 195438

You can prepend 'https://www.example.com' in your first loop, for example:

links = []
for link in soup.find_all("a", {"class": "product-info__caption"}):
    links.append('https://www.example.com' + link.attrs['href'])

for x in links:
    # your next stuff here
    # ...

Upvotes: 1

Related Questions