Reputation: 4212
I'm screen scraping using Curl like this:
<?php
$url = "http://www.bbc.com/news/";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
echo $curl_scraped_page;
?>
And then I echo the content on an HTML page. The problem is that when I look in my console I see 404 errors because of relative URLs assigned to javascript files. For instance if the URL is: somejavascriptfile.js
on loading the page my domain name is added like so: http://mydomain/somejavascriptfile.js
These paths are obviously not correct.
So what can I do to get the actual URL of the js file instead? If the URLs where in the body I could use jQuery (split/replace) to alter but this wouldn't work in this case.
Upvotes: 1
Views: 133
Reputation: 276296
You can add a base
tag to the scraped HTML.
Open it with an HTML parser like tidy, go to the start of the head section and append a <base>
tag. A base tag will redirect all resource access to a known location .
Upvotes: 4