CODEWITHSUNDEEP
pythonseleniumweb-scraping
flow.v
flow.v

Reputation: 15

Get URL in Selenium Python

I am very new to python and I am looking to scrape following website:Link

I think that Selenium might be the right tool and I started to write following code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

path='http://planning.hackney.gov.uk/Northgate/PlanningExplorer/generalsearch.aspx'

browser = webdriver.Firefox()
browser.get(path)

elem = browser.find_element_by_id('txtPostCode')
elem.clear()
elem.send_keys("E9 7JP")
elem.send_keys(Keys.RETURN)

print (browser.current_url)

So far so good, it works. However, the return value of browser.current_url is not quite what is displayed in the url bar of my browser. I mean the the return value of the script is:

//planning.hackney.gov.uk/Northgate/PlanningExplorer/generalsearch.aspx

however the url in the browser is showing me this one here:

//planning.hackney.gov.uk/Northgate/PlanningExplorer/Generic/StdResults.aspx?PT=Planning%20Applications%20On-Line&SC=Postcode%20is%20E9%207JP&FT=Planning%20Application%20Search%20Results&XMLSIDE=/Northgate/PlanningExplorer/SiteFiles/Skins/Hackney/Menus/PL.xml&XSLTemplate=/Northgate/PlanningExplorer/SiteFiles/Skins/Hackney/xslt/PL/PLResults.xslt&PS=10&XMLLoc=/Northgate/PlanningExplorer/Generic/XMLtemp/j5jzxiwxklgslnam4qffypw5/052dd052-3993-4f10-83aa-dd0c6c326676.xml

Now I wonder how to get hold of this adress?!

Thanks a lot!

Upvotes: 1

Views: 14039

Answers (1)

demouser123
demouser123

Reputation: 4264

Did you made any other request in between checking your script returned URL and the URL shown by the browser. The request URL sent post the Keys.RETURN adds a session identifier with the URL, which might be the reason why you are getting different URL.

I have this script 

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
chromepath='chrome_driver_path' //change this to your chromedriver path
driver = webdriver.Chrome(chromepath)

driver.get('http://planning.hackney.gov.uk/Northgate/PlanningExplorer/generalsearch.aspx')

print(driver.current_url)

elem = driver.find_element_by_id('txtPostCode')
elem.clear()
elem.send_keys("E9 7JP")
elem.send_keys(Keys.RETURN)

print (driver.current_url)

driver.quit()

Keypress code has been copied from your code itself. I get an identical URL from both the browser and the script

Script gives me this URL - Link Browser gives me this same URL - Copied Manually

Upvotes: 3

Related Questions