CODEWITHSUNDEEP
pythonweb-scrapingmechanize
vgoklani
vgoklani

Reputation: 11756

Problems using python mechanize with Google Searches

I recently discovered the python library mechanize, which I would like to use to get to get links from Google searches, but can't make sense of the output. Here is my code snippet:

import mechanize, cookielib
br = mechanize.Browser()
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')] 
br.set_handle_robots(False)
url = 'https://www.google.com/search?num=10&hl=en&site=&q=dog&oq=dog&aq=f&aqi=g10&aql=1&gs_sm=e'

response = br.open(url)

links = [link for link in br.links()]

which runs correctly, but the output looks like this:

[
Link(base_url='https://www.google.com/search?num=10&hl=en&site=&q=dog&oq=dog&aq=f&aqi=g10&aql=1&gs_sm=e', url='/support/websearch/bin/answer.py?answer=186645&form=bb&hl=en', text='Learn more', tag='a', attrs=[('href', '/support/websearch/bin/answer.py?answer=186645&form=bb&hl=en')]),
Link(base_url='https://www.google.com/search?num=10&hl=en&site=&q=dog&oq=dog&aq=f&aqi=g10&aql=1&gs_sm=e', url='http://www.google.com/intl/en/options/', text='More', tag='a', attrs=[('class', 'gbgt'), ('id', 'gbztm'), ('href', 'http://www.google.com/intl/en/options/'), ('onclick', 'gbar.tg(event,this)'), ('aria-haspopup', 'true'), ('aria-owns', 'gbd')]),
Link(base_url='https://www.google.com/search?num=10&hl=en&site=&q=dog&oq=dog&aq=f&aqi=g10&aql=1&gs_sm=e', url='/webhp?hl=en&tab=ww', text='', tag='a', attrs=[('href', '/webhp?hl=en&tab=ww'), ('onclick', 'gbar.logger.il(39)'), ('title', 'Go to Google Home')]),
...,
]

how do I get the actual URL, instead of this "click-me" style response?

Thanks!

Upvotes: 1

Views: 585

Answers (1)

Matthew Trevor
Matthew Trevor

Reputation: 14961

You're pulling in every link on the page, you need to filter it down to just the relevant search result links. I think this will do what you want:

links = [link for link in br.links() if any(attr==('class','l') for attr in link.attrs)]

The main search result links all appear to have class=l as an attribute. I'm not familiar enough with mechanize to know if you could do this in the links() call.

Upvotes: 2

Related Questions