php curl post not processing, acting like get with no data

Question

I am trying to use php's curl to post to a sites form for me then extract the result, but it is not working. Instead it shows a blank form, like I just did a basic GET reequest to the page.

<?php
$domains = [
    'expireddomains.net',
    'stackoverflow.com',
    'toastup.com'
];

$ccd = '';
foreach ($domains as $domain) {
    $ccd .= $domain . '\r\n';
}

// set post fields
$post = [
    'removedoubles' => '1',
    'removeemptylines' => '1',
    'showallwordmatches' => '1',
    'wordlist' => 'en-v1',
    'camelcasedomains' => $ccd,
    'button_submit' => 'Camel+Case+Domains'
];

$ch = curl_init('https://www.expireddomains.net/tools/camel-case-domain-names/');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);

$headers = [
    'Referer: https://www.expireddomains.net/tools/camel-case-domain-names/',
    'Content-Type: application/x-www-form-urlencoded',
    'Origin: https://www.expireddomains.net'
];

curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

// execute!
$response = curl_exec($ch);

// close the connection, release resources used
curl_close($ch);

I confirmed the postdata formatting and names using the network tab in the browser's dev tools to check the request.

Originally I wasn't sending any headers, then I thought maybe the site validated the origin or referer, but even adding that didn't work.

I checked, the form doesn't include any hidden fields for something like a CSRF token or anything.

Any ideas?

Answer 1

For application/x-www-form-urlencoded, use http_build_query and let it encode the values like +'s etc, plus the seperator between domains is | not new lines.

<?php
$domains = [
    'expireddomains.net',
    'stackoverflow.com',
    'toastup.com'
];

// set post fields
$post = [
    'removedoubles' => 1,
    'removeemptylines' => 1,
    'showallwordmatches' => 1,
    'wordlist' => 'en-v1',
    'camelcasedomains' => implode(' | ', $domains),
    'button_submit' => 'Camel Case Domains'
];

$ch = curl_init('https://www.expireddomains.net/tools/camel-case-domain-names/');

$headers = array();
$headers[] = 'authority: www.expireddomains.net';
$headers[] = 'pragma: no-cache';
$headers[] = 'cache-control: no-cache';
$headers[] = 'origin: https://www.expireddomains.net';
$headers[] = 'upgrade-insecure-requests: 1';
$headers[] = 'dnt: 1';
$headers[] = 'content-Type: application/x-www-form-urlencoded';
$headers[] = 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36';
$headers[] = 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9';
$headers[] = 'sec-fetch-site: same-origin';
$headers[] = 'sec-fetch-mode: navigate';
$headers[] = 'sec-fetch-user: ?1';
$headers[] = 'sec-fetch-dest: document';
$headers[] = 'referer: https://www.expireddomains.net/tools/camel-case-domain-names/';
$headers[] = 'referrer-policy: same-origin';
$headers[] = 'accept-language: en-GB,en-US;q=0.9,en;q=0.8';
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate');

// execute!
$response = curl_exec($ch);

// close the connection, release resources used
curl_close($ch);

// parse whats in textarea
$dom = new DOMDocument();
libxml_use_internal_errors(true);
$dom->loadHTML($response);
libxml_clear_errors();

$result = [];
foreach ($dom->getElementsByTagName('textarea') as $textarea) {
    if ($textarea->getAttribute('name') === "camelcasedomains") {
        $result = explode(' | ', $textarea->nodeValue);
    }
}

print_r($result);

Result:

Array
(
    [0] => ExpiredDomains.net
    [1] => ExpiredDoMains.net
)

You could probably remove most of the headers, if not needed. I just added them all to exactly match the request, but ended up being the aforementioned encoding.