CODEWITHSUNDEEP
javascriptpythonbots
Valachio
Valachio

Reputation: 1135

Building a bot that fetches data from browser and saves it as text file

My problem is a bit complex. Hard to explain with words, so I broke it down into steps with pictures at each step.

  1. Select a single date from these boxes. Hit submit

  2. I will land on a page with a table. Copy the <tbody> element from the developer console.

  3. Paste it into a text file. Save the text file with the date that was selected.

  4. Repeat steps 1-3 for as many times as needed, selecting a new date each time (01-15-2018, 01-14-2018, 01-13-2018, and so on...)

Is it even possible to build a bot that does this? If yes, what tools would I use?

I know a fair amount of JavaScript and Python, so I'd prefer to use those 2 if possible.

Upvotes: 0

Views: 549

Answers (2)

John R
John R

Reputation: 1508

Would need to know the URL you're looking at/look at the page source. If the date is supplied as any part of a request, and the response contains this data you're looking for, it should be simple to farm and analyze that data from a python script.

Walk through your clicks with the network tab of your browser's developer tools and you should see a request go out when you hit submit. Expedia just uses query parameters, and so the entire URL that you'll need pops up in the URL bar of your browser after hitting submit...

Tools:

If request-based: Python Requests module

If something cached/more complicated, there are tools for automating clicks and saving the results...I would guess that this won't be necessary though...

Update:

AJAX calls are HTTP requests and responses, and so you should be able to observe them in your networking tab of our web browser developer tools, and then mimic that request from a script, rather than from your browser.

The readability of the requests/responses and/or any measures the organization has implemented to make any application other than a browser unable to get the same response would be potential impediments, but even those should be imitable. If your browser is making the request, then there is no reason your python script can't make the same.

The method you seem to be interested in, although it sounds more complicated to me, is possible with automation tools like Selenium, as the other poster answered. Best of luck.

Upvotes: 1

Karan Shishoo
Karan Shishoo

Reputation: 2802

It is possible:

Take a look at selenium library (its commonly used for automated testing) for python. It should be able to select single dates, hit the submit button then go through the HTML code and grab data in the tag. After that you can use python by itself to store this data in a text file with a name of your choice in a location of your choice.

Upvotes: 0

Related Questions