CODEWITHSUNDEEP
pythonhtmlurlget
gaming_enthusiast
gaming_enthusiast

Reputation: 125

Sending GET Request with fragment identifier in Python

I'm trying to send a GET request using the requests module including a fragment identifier. I have the following code:

url = 'http://steamcommunity.com/market/search?appid=730#p20_quantity_desc'
page = requests.get(url, headers=headers)

However I always end up getting the base page (http://steamcommunity.com/market/search?appid=730) instead of the page with the fragment identifier (the #p20_quanitity_description doesn't seem to be sent).

urllib2 also isn't working for me with the code:

req = urllib2.Request(url, headers={ 'User-Agent': 'Mozilla/5.0' })
page = urllib2.urlopen(req).read().decode('UTF-8', 'ignore')

How would I go about sending a GET request and including the #p20_quanitity_description in the URL?

Upvotes: 4

Views: 2080

Answers (1)

TkTech
TkTech

Reputation: 4976

The anchor (p20_quantity_desc) means nothing to the server. There's a bit of Javascript on the page that's changing the sort order of the results based on this anchor, but this is client-side. Requests/urllib will see the same page response with or without the anchor.

Try disabling Javascript on the page and you'll see what I mean.

What you want to do instead is make a request to the API endpoint that the page is using. Here's an example:

http://steamcommunity.com/market/search/render/?query=&start=0&count=10&search_descriptions=0&sort_column=quantity&sort_dir=asc&appid=730

Note the sort_column argument? This is the value you can change to decide the ordering of the results. Use a library like lxml to parse the results_html field and presto, you're done.

Upvotes: 6

Related Questions