PHP cURL works with session id, but it doesn't work with cookiejar

Question

I've been trying to get some page data from this website using cURL. Page is behind authorization, using cookies, without SSL.
I checked a lot of manuals and examples of setting up php cURL script, but none seemed to be working.

Every time I run my script, cookie file updates, but I get empty string as a result. If I set CURLOPT_FOLLOWLOCATION to 1, I get login page as a result. So I assume, original script returns redirect back to login page.
I tried messing with CURLOPT_USERAGENT, CURLOPT_REFERER, but it didn't help.

Also, if I manually set CURLOPT_COOKIE, PHPSESSID (from real login session using browser and human input), it works fine.

So, here's my code:

<?php
set_time_limit(10);
define('USERNAME', 'username');
define('PASSWORD', 'password');
define('USER_AGENT', 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36');
define('COOKIE_FILE', 'cookie.txt');
define('LOGIN_FORM_URL', 'http://website/auth');
define('LOGIN_ACTION_URL', 'http://website/distribution/index');
$postValues = array(
    'login_msisdn' => USERNAME,
    'password' => PASSWORD
);
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, LOGIN_FORM_URL);
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_POSTFIELDS, http_build_query($postValues));
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_COOKIEJAR, realpath(COOKIE_FILE));
//curl_setopt($curl, CURLOPT_COOKIE, "PHPSESSID=relkdrgg94gfdgfg834g");
curl_setopt($curl, CURLOPT_USERAGENT, USER_AGENT);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_REFERER, LOGIN_FORM_URL);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 0);
curl_exec($curl);
if(curl_errno($curl)){
    throw new Exception(curl_error($curl));
}
curl_close($curl);

$curl = curl_init()
curl_setopt($curl, CURLOPT_URL, LOGIN_ACTION_URL);
curl_setopt($curl, CURLOPT_COOKIEFILE, realpath(COOKIE_FILE));
curl_setopt($curl, CURLOPT_USERAGENT, USER_AGENT);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$html = curl_exec($curl);
echo $html;

Answer 1

You are only saving the cookie (using CURLOPT_COOKIEJAR) from your first curl call, but not loading during your second curl call. That's why no cookie is used during second call. Use the following with your second curl call.

curl_setopt($curl, CURLOPT_COOKIEFILE, realpath(COOKIE_FILE));

Secondly, you have to close your curl hand and initialize it again before making second curl request. The curl option CURLOPT_COOKIEJAR helps to save cookie on file, but it does it when the curl handle is closed.

curl_close($curl);
$curl = curl_init();
// here goes the second one
curl_setopt($curl, CURLOPT_URL, LOGIN_ACTION_URL);

Also, as suggested from the comment (that I've missed), enable the option CURLOPT_RETURNTRANSFER so that the curl returns output.

curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);

Another note, please use full path for the cookie file. Otherwise it may not work when you run from browser (also make sure you have write permission on cookie file).

define('COOKIE_FILE', '/some/directory/cookie.txt');

Answer 2

/*
    1) Make first request in main page and after do the login
    2) I added some headers
    3) Check if are all parameters in post ( ex: "&login=Submit" )
    4) If is basic authorization use curl_setopt($curl, CURLOPT_USERPWD, "$username:$password");
    5) Debug header/ errors ...
*/

$url1 = "http://website/";
$url2 = "http://website/auth";
$url3 = "http://website/distribution/index";

$user = "username";
$pass = "password";

$post = "user=".$user."&pass=".$pass;


get_url($url1,'',$url1);

$login = get_url($url2,$post,$url1);

$data = get_url($url3,'',$url1);

print_r($data);



function get_url($url,$post,$refer) {
    $ssl = substr(strtolower($url),0,8)=='https://' ? true : false;
    $cookie = getcwd().DIRECTORY_SEPARATOR.'cookie.txt';
    $header[0] = "text/xml,application/xml,application/xhtml+xml,";
    $header[0] .= "text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
    $header[] = "Cache-Control: max-age=0";
    $header[] = "Connection: keep-alive";
    $header[] = "Keep-Alive: 300";
    $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
    $header[] = "Accept-Language: en-us,en;q=0.5";
    $header[] = "Pragma: ";
    $agent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36";
    $refer = !empty($refer) ? $refer : "http://www.google.com/";
    $curl = curl_init();
    curl_setopt($curl, CURLOPT_URL, $url);
    curl_setopt($curl, CURLOPT_USERAGENT, $agent);
    if( !empty($post) ) {
        curl_setopt($curl, CURLOPT_POST, 1);
        curl_setopt($curl, CURLOPT_POSTFIELDS, $post);
    }
    if( $ssl ) {
        curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
    }
    curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
    curl_setopt($curl, CURLOPT_REFERER, $refer);
    curl_setopt($curl, CURLOPT_COOKIEJAR, $cookie);
    curl_setopt($curl, CURLOPT_COOKIEFILE, $cookie);
    curl_setopt($curl, CURLOPT_AUTOREFERER, true);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($curl, CURLOPT_TIMEOUT,60);

    $html = curl_exec($curl);
    $info = curl_getinfo($curl);
    $error = '';
    if( $html === false ) {
        $error = 'Curl error: ' . curl_error($curl);
    }               
    curl_close($curl);
    $arr = array();
    $arr['html'] = $html;
    $arr['info'] = $info;
    $arr['error'] = $error;
    return $arr;    

}   

Answer 3

For save and use cookie file with CURL, i use this code:

$ckfile = tempnam('/tmp', 'CURLCOOKIE');
curl_setopt($curl, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt($curl, CURLOPT_COOKIEFILE, $ckfile);