CODEWITHSUNDEEP
javascriptpythonweb-scrapingdopostback
Dance Party2
Dance Party2

Reputation: 7536

Python Web Scraping with JavaScript Do Postback

I have been trying to:

  1. Go to: mdoe.state.mi.us/moecs/PublicCredentialSearch.aspx

  2. Enter a certificate number (for the sake of illustration, you can just search for "Davidson" as the last name).

  3. Click on a link corresponding to "Professional Teaching Certificate".

  4. Copy and paste the resulting table.

The rub seems to be with the JavaScript doPostBack() part, as it requires rendering, I believe, to get the data.

When viewing the source code, see how the href part identifies an individual link like this? (for the 6th link down): 

href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$gViewCredentialSearchList$ctl07$link1','')

From this:

<td class="MOECSNormal" style="border-color:Black;border-width:1px;border-style:Solid;">Professional Teaching Certificate Renewal</td><td class="MOECSNormal" style="border-color:Black;border-width:1px;border-style:Solid;">
                                            <a id="ContentPlaceHolder1_gViewCredentialSearchList_link1_5" ItemStyle-BorderColor="Black" ItemStyle-BorderStyle="Solid" ItemStyle-BorderWidth="1px" href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$gViewCredentialSearchList$ctl07$link1','')">CC-XWT990004102</a>
                                        </td>

I'm looking for a way (via Python) to get the data I need into a table, given a certification number and certificate name (i.e. "Professional Teaching Certificate".

I have tried following a tutorial using PyQt4, but installing it alone was traumatic.

Thanks in advance!

Upvotes: 0

Views: 1170

Answers (1)

johnII
johnII

Reputation: 1433

You can open the page in a browser e.g. Chrome and study how the interaction is done between the page and the server, normally this information can be seen in the network tab of Developer tool, this way you can formulate a python script to do the steps maybe using requests library

or

You can use selenium-python to do simulate your browser interaction (including javascript calls) until you got to the page where your interested data belongs to.

Upvotes: 0

Related Questions