CODEWITHSUNDEEP
pythonmechanize
Bhaxy
Bhaxy

Reputation: 5574

Using Python mechanize on websites that use DHTML, AJAX, etc.?

So, let's say I'm trying to create something that replies to tweets of a certain "hashtag keyword" on twitter (for example "#FirstWorldProblems") I have a script that looks like this:

# apply settings, create a mechanize.Browser, etc.

login() # log into twitter

# at this point we've logged into twitter, now, we will perform navigate to their search page and run a search query:
br.open('http://twitter.com/search?q=' + hashtag)
print(br.response().read()) # print the response

So, what I have above is sort of an abbreviated version to quickly get to the spot giving me trouble.

I set up a browser, log into twitter, all done no problemo. But, then I run a search for the hashtag (using br.open) and then I print the response.

On Twitter, the "Reply" link only appears when you hover over a specific link and leads to "#" (because it opens a little pop-up thing where you can enter your reply), how would I click on the "Reply" link, because it doesn't show up in the response?

Upvotes: 0

Views: 570

Answers (1)

synthesizerpatel
synthesizerpatel

Reputation: 28036

If your problem is actually just accessing Twitter, dmedvinsky is probably right.

However, if you really want to be able to scrape websites (while allowing their javascript to run as it normally would..) you'll probably want something a bit more robust.

While it's a lot of baggage, I strongly urge you to grab Qt, PySide, and get familiar with QWebKit. You can drive a 'real' web browser from Python and get all the benefits (and problems;) one would expect. But, so far it's the best and cleanest method I've found to do what you're asking about.

Upvotes: 2

Related Questions