CODEWITHSUNDEEP
apiweb-scraping
user3214460
user3214460

Reputation: 1

Scraping and non-API interaction

I'm considering building an application to check stock levels of various stores. Some of the stores do not have an API that exposes this information.

I'm considering building a scraper for each website.

Do you think this will work? Can I build just one script to check all sites? For example, this is the markup from one site:

<div class = "stock status6">
  <b> In stock </b>
</div>

So my script would need to parse this to extract the stock information.

From what I understand, the problem with this technique is that if a site changes markup then my scraper could stop working.

How can I get the benefits of working with an API when one isn't available? Can Javascript help here?

Upvotes: 0

Views: 48

Answers (1)

Charlie Egan
Charlie Egan

Reputation: 5203

I'm considering building a scraper for each website.

This is your only option when you don't have access to an API.

Do you think this will work? Can I build just one script to check all sites?

Scraping is a very common approach to such problems. How you structure the script is a detail here, however each site will likely need to be parsed in a different way.

From what I understand, the problem with this technique is that if a site changes markup then my scraper could stop working.

Yes, you need to be ready to update your script when the page changes or get the data from another source if it's made unavailable.

How can I get the benefits of working with an API when one isn't available? Can Javascript help here?

Scraping can be a hard to maintain and is rarely as reliable as an API.

Client-side Javascript can't be used to scrape data because of the Same-origin policy. Javascript can be used to build a scraper if ran from the server.

Upvotes: 0

Related Questions