CODEWITHSUNDEEP
python-2.7web-crawlerdata-acquisition
Chandrashekar Kuyyamudi
Chandrashekar Kuyyamudi

Reputation: 1

Script to crawl through different pages and acquire data

I am planning to do a network analysis of bmtc bus connectivity network... So i need to acquire data regarding bus routes. The best website as far as i know is http://www.narasimhadatta.info/bmtc_query.html Under the "search by route " option the whole list of routes is given and one can select any one of them and on clicking "submit" it displays the detailed route . Previously when I acquired data online I used to encash upon the fact that each item (in this case route number) lead to distinct URL, and I used to acquire data from the source page using Python. But here irrespective of the bus route the final page always has the URL http://www.narasimhadatta.info/cgi-bin/find.cgi and it's source page doesn't contain the route details!

I am only comfortable with Python and Matlab. I couldn't figure out any means to acquire data from that website. If you can see something, technically one should be able to download the data (at least thats what I believe). So can you please help me out with a code which crawls through each bus route number automatically and downloads the route details?

Upvotes: 0

Views: 360

Answers (1)

shjain
shjain

Reputation: 41

I looked at the url you mentioned. if you have a list of route numbers, you can use the following url sturcture to extract data.

http://www.narasimhadatta.info/cgi-bin/find.cgi?route=270S

or

http://www.narasimhadatta.info/cgi-bin/find.cgi?route=[route number from you list]

Upvotes: 1

Related Questions