CODEWITHSUNDEEP
python
Yasser Hussain
Yasser Hussain

Reputation: 73

Using BeautifulSoup, trying to extract data within an options tags

I am trying to build a sneaker bot, right now i am struggling to obtain data values from the site, these are used in the url for the checkout page.

I've used the following BeautifulSoup command to scrape

size=soup.find(id="custom-variants")

and got the following output:

<select id="custom-variants">
<option disabled="disabled" selected="selected">Make a choice</option>
<option data-status="available" data-value="191707602">Size : EU 38.5 (US 6)</option>
<option data-status="available" data-value="191707614">Size : EU 39 (US 6.5)</option>
<option data-status="available" data-value="191707617">Size : EU 40 (US 7)</option>
<option data-status="available" data-value="191707623">Size : EU 40.5 (US 7.5)</option>
<option data-status="available" data-value="191707626">Size : EU 41 (US 8)</option>
<option data-status="available" data-value="191707632">Size : EU 42 (US 8.5)</option>
<option data-status="available" data-value="191707641">Size : EU 42.5 (US 9)</option>
<option data-status="available" data-value="191707644">Size : EU 43 (US 9.5)</option>
<option data-status="available" data-value="191707650">Size : EU 44 (US 10)</option>
<option data-status="available" data-value="191707656">Size : EU 44.5 (US 10.5)</option>
<option data-status="available" data-value="191707659">Size : EU 45 (US 11)</option>
<option data-status="available" data-value="191707662">Size : EU 45.5 (US 11.5)</option>
<option data-status="available" data-value="191707665">Size : EU 46 (US 12)</option>
<option data-status="available" data-value="191707599">Size : EU 38 (US 5.5)</option>
</select>

But what i really want is the data-value from within the option tag.

Can anyone help please.

Upvotes: 0

Views: 251

Answers (1)

balderman
balderman

Reputation: 23825

Try this

from bs4 import BeautifulSoup


html = '''<select id="custom-variants">
<option disabled="disabled" selected="selected">Make a choice</option>
<option data-status="available" data-value="191707602">Size : EU 38.5 (US 6)</option>
<option data-status="available" data-value="191707614">Size : EU 39 (US 6.5)</option>
<option data-status="available" data-value="191707617">Size : EU 40 (US 7)</option>
<option data-status="available" data-value="191707623">Size : EU 40.5 (US 7.5)</option>
<option data-status="available" data-value="191707626">Size : EU 41 (US 8)</option>
<option data-status="available" data-value="191707632">Size : EU 42 (US 8.5)</option>
<option data-status="available" data-value="191707641">Size : EU 42.5 (US 9)</option>
<option data-status="available" data-value="191707644">Size : EU 43 (US 9.5)</option>
<option data-status="available" data-value="191707650">Size : EU 44 (US 10)</option>
<option data-status="available" data-value="191707656">Size : EU 44.5 (US 10.5)</option>
<option data-status="available" data-value="191707659">Size : EU 45 (US 11)</option>
<option data-status="available" data-value="191707662">Size : EU 45.5 (US 11.5)</option>
<option data-status="available" data-value="191707665">Size : EU 46 (US 12)</option>
<option data-status="available" data-value="191707599">Size : EU 38 (US 5.5)</option>
</select>'''


soup = BeautifulSoup(html, 'html.parser')
for option in soup.find_all('option'):
    print(option.attrs)
    print(option.text)
    print('-----')
data_values = [option.attrs.get('data-value') for option in soup.find_all('option')]
print(data_values)

Upvotes: 1

Related Questions