CODEWITHSUNDEEP
web-scrapingheadless
Nate Thompson
Nate Thompson

Reputation: 395

Is there a way to retrieve data from a website/app that doesn't offer open APIs

Let's say, for example, I'm building an application where a user searches for a shirt based on it's UPC code. The application's job is to find that shirt at other various retailers. Retailer A has a API that can be easily used to locate product. However, retailer B doesn't have an API open to developers.

How would a developer handle retailer B?

The ideas that I've come up with were:

Has anyone encountered this before, or does anyone know of an existing technology or method to approach this issue? I've never understood how you would handle data retrieval without using APIs.

Upvotes: 0

Views: 328

Answers (1)

mankowitz
mankowitz

Reputation: 2051

Here is something I did in php using curl. It should help get you started. 

$url = 'https://newjersey.xxxxxx.net/login';
$cookie="cookie.txt";

$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36");
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt ($ch, CURLOPT_REFERER, $url);
curl_setopt ($ch, CURLOPT_VERBOSE, true);

$result = curl_exec ($ch);
preg_match('/meta content="([^"]+)" name="csrf-token"/', $result, $matches);
$token = $matches[1];
$creds['authenticity_token'] = $token;

print "login page with token $token\n";

curl_setopt ($ch, CURLOPT_URL, 'https://newjersey.xxxxx.net/auth/identity/callback');
curl_setopt ($ch, CURLOPT_POSTFIELDS, http_build_query($creds));
curl_setopt ($ch, CURLOPT_POST, 1);
$result = curl_exec ($ch);

Once logged in, you can get use a regex to get the data you are looking for.

Upvotes: 0

Related Questions