CODEWITHSUNDEEP
phpcurlweb-scrapingweb-crawler
dharanbro
dharanbro

Reputation: 1342

cant set Host in CURL PHP

I am unable to set the host in curl. It still shows as localhost if i use the following code

function wget($url)
        {

            $agent= 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1';
            $curlHeaders = array (
                    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
                    'Accept-Encoding: gzip, deflate',
                    'Accept-Language: en-US,en;q=0.5',
                    'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1',
                    'Connection: Keep-Alive',
                    'Pragma: no-cache',
                    'Referer: http://example.com/',
                    'Host: hostname',
                    'Cache-Control: no-cache',
                    'Cookie: visid_incap_185989=9v1q8Ar0ToSOja48BRmb8nn1GFUAAAAAQUIPAAAAAABCRWagbDIfmlN9NTrcvrct; incap_ses_108_185989=Z1orY6Bd0z3nGYE2lbJ/AXn1GFUAAAAAmb41m+jMLFCJB1rTIF28Mg==; _ga=GA1.3.637468927.1427699070; _gat=1; frontend=rqg7g9hp2ht788l309m7gk8qi7; _gat_UA-1279175-12=1; __utma=233911437.637468927.1427699070.1427699078.1427699078.1; __utmb=233911437.2.10.1427699078; __utmc=233911437; __utmz=233911437.1427699078.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt_UA-1279175-1=1; _cb_ls=1; _chartbeat2=S0WVXDwMWnCFBgQp.1427699081322.1427699232786.1; PRUM_EPISODES=s=1427699568560&r=http%3A//example.com/'

            );
            $ch = curl_init();
            curl_setopt ($ch, CURLOPT_HTTPHEADER, $curlHeaders);
            curl_setopt ($ch, CURLOPT_HEADER, TRUE);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
            curl_setopt($ch, CURLOPT_USERAGENT, $agent);
            curl_setopt($ch, CURLOPT_URL,$url);
            $result=curl_exec($ch);
            return $result;
        }

I use fiddler to track the network requests. where I found the host is still as localhost enter image description here

If I load this same Link in browser i get as following in fiddler

enter image description here

I need my specified domain to be accessed. How can I achieve this? Note: I am aware that host name should not contain the protocol.

Alternatively

Also i would like to know is it possible to get the source code of a website the could be seen in browser through terminal?

Upvotes: 10

Views: 11632

Answers (5)

Drakes
Drakes

Reputation: 23660

Assuming we are not trying spoof the Host header, omit the Host header altogether and let curl sort it out. In this case, just remove 'Host: hostname', because you already get curl to automatically set this with your code near the bottom with curl_setopt($ch, CURLOPT_URL, $url);.

If you really want to set the Host header yourself, then just replace

'Host: hostname',

with

"Host: ". parse_url($url, PHP_URL_HOST),

(Note: This function doesn't work with relative URLs.)

Upvotes: 6

yergo
yergo

Reputation: 4980

According to HTTP quick specification read, I assume your problems are happening because of improper Host header being send. I was able to download some websites with following code:

function wget($url, $follow = true) {

    $host = parse_url($url);

    $agent       = 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1';
    $curlHeaders = array(
        'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
        'Accept-Encoding: gzip, deflate',
        'Accept-Language: en-US,en;q=0.5',
        'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1',
        'Connection: Keep-Alive',
        'Pragma: no-cache',
        'Referer: http://example.com/',
        'Host: ' . $host['host'] . (isset($host['port']) ? ':' . $host['port'] : null), // building host header
        'Cache-Control: no-cache',
        'Cookie: visid_incap_185989=9v1q8Ar0ToSOja48BRmb8nn1GFUAAAAAQUIPAAAAAABCRWagbDIfmlN9NTrcvrct; incap_ses_108_185989=Z1orY6Bd0z3nGYE2lbJ/AXn1GFUAAAAAmb41m+jMLFCJB1rTIF28Mg==; _ga=GA1.3.637468927.1427699070; _gat=1; frontend=rqg7g9hp2ht788l309m7gk8qi7; _gat_UA-1279175-12=1; __utma=233911437.637468927.1427699070.1427699078.1427699078.1; __utmb=233911437.2.10.1427699078; __utmc=233911437; __utmz=233911437.1427699078.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt_UA-1279175-1=1; _cb_ls=1; _chartbeat2=S0WVXDwMWnCFBgQp.1427699081322.1427699232786.1; PRUM_EPISODES=s=1427699568560&r=http%3A//example.com/'
    );
    $ch          = curl_init();
    curl_setopt($ch, CURLOPT_HTTPHEADER, $curlHeaders);
    curl_setopt($ch, CURLOPT_HEADER, TRUE);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, $follow); // following redirects or not
    curl_setopt($ch, CURLOPT_USERAGENT, $agent);
    curl_setopt($ch, CURLOPT_URL, $url);
    $result      = curl_exec($ch);
    return $result;
}

echo(wget('http://example.com'));

Anyway this function is not an universal build. Personally I would add saving cookies between redirection requests etc. Essential change is within 'Host' header line. I'm building there proper Host header based on full $url provided to function.

Upvotes: 1

Francisco Félix
Francisco Félix

Reputation: 2413

Set the full URL into CURLOPT_URL.

Upvotes: -1

Waqas Shakeel
Waqas Shakeel

Reputation: 88

If you are using windows and xampp then try to use virtual host rather than localhost, then it will start working, I did the same.

Upvotes: 1

Ayyanar G
Ayyanar G

Reputation: 1545

try like this,

  curl_init('XXX.XXX.XXX.XXX');
  curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: subdomain.hostname.com'));

Upvotes: 2

Related Questions