CODEWITHSUNDEEP
asp.nethtmlparsingscreen-scrapinghtml-content-extraction
Mike
Mike

Reputation:

Parse a .Net Page with Postbacks

I need to read data from an online database that's displayed using an aspx page from the UN. I've done HTML parsing before, but it was always by manipulating query-string values. In this case, the site uses asp.net postbacks. So, you click on a value in box one, then box two shows, click on a value in box 2 and click a button to get your results.

Does anybody know how I could automate that process?

Thanks,

Mike

Upvotes: 0

Views: 2336

Answers (3)

Jon Galloway
Jon Galloway

Reputation: 53125

I'd look at HtmlAgilityPack with the FormProcessor addon.

Upvotes: 1

ConsultUtah
ConsultUtah

Reputation: 6809

Watin would be my first choice. You would code the selecting and clicking, then parse the HTML after.

Upvotes: 1

Joel Coehoorn
Joel Coehoorn

Reputation: 415860

You may still only need to send one request, but that one request can be rather complicated. ASP.Net is notoriously difficult (though not impossible) to screen scrape. Between event validation and the ViewState, it's tricky to get your requests just right. The simplest way to do it is often to use a sniffer tool like fiddler to see exactly what the http request looks like, and then just mimic that request.

If you do still need to send two requests, it's because the first request also places some state in a session somewhere, and that means whatever you use to send those requests needs to be able to send them with the same session. This often means supporting cookies.

Upvotes: 1

Related Questions