CODEWITHSUNDEEP
pythonbeautifulsouppython-requests
Mike
Mike

Reputation: 496

Select using requests python

Using BeautifulSoup and Requests:

Input: 

select_object=soup.find('select',{'id':'limit'})
print(select_object)

Output:

<select id="limit" name="limit" class="inputbox" size="1" onchange="this.form.submit()">
<option value="5">5</option>
<option value="10">10</option>
<option value="15">15</option>
<option value="20" selected="selected">20</option>
<option value="25">25</option>
<option value="30">30</option>
<option value="50">50</option>
<option value="100">100</option>
<option value="0">All</option>

What i need to do is move the select="selected" to the last option, All.

What i tried:

for item in select_object:
    if str(item) == '<option selected="selected" value="20">20</option>':
        item.replace_with('<option value="20">20</option>')
    if str(item) == '<option value="0">All</option>':
        item.replace_with('<option selected="selected" value="0">All</option>')
request=requests.post(url,params=select_object)
soup=BeautifulSoup(request.content)

The above does not work because instead of < and > i get &lt

I also tried:

for item in select_object:
    if str(item) == '<option selected="selected" value="20">20</option>':
        new_tag2=BeautifulSoup('<option value="20">20</option>')
        new_tag2=new_tag2.html.body.contents[0]
        item.replace_with(new_tag2)
    if str(item) == '<option value="0">All</option>':
        newtag2=BeautifulSoup('<option selected="selected" value="0">All</option>')
        newtag2=newtag2.html.body.contents[0]
        item.replace_with(newtag2) value="0">All</option>')
request=requests.post(url,params=select_object)
soup=BeautifulSoup(request.content)

This works, if i search for the select item again it's the correct one but there is no change in the page, i only get the first 20 results, instead of getting all of them. Any ideas? I think i'm not posting right or maybe there is a simpler way of doing it. I don't think params=select_object is right but...

Upvotes: 0

Views: 4461

Answers (2)

user11909331
user11909331

Reputation: 1

For 'post' you should use 'data' instead of 'params'. Use 'params' for 'get'.

Upvotes: 0

CtheSky
CtheSky

Reputation: 2624

replace_with will escape the string, but it also receives a tag so you can use:

new_tag = BeautifulSoup('<option value="20">20</option>').option
item.replace_with(new_tag)

If you only want to change the attribute, it's easier to manipulate it directly:

for item in select_object:
    if str(item) == '<option selected="selected" value="20">20</option>':
        del item['selected']
    if str(item) == '<option value="0">All</option>':
        item['selected'] = 'selected'

Update:

You need to send a post request with your form data, you need to get the url and prepare the data. Something like:

# get the url
action = soup.find('form', ...).get('action')
post_url = get_post_url(host, action)

# parse the html and prepare the form
form = {'limit': '0', ...}

# send post request the form data
response = requests.post(post_url, data=form)

Upvotes: 2

Related Questions