Navigates to URL but gets no response

Question

I'm using casperjs and I had an array of links and I was opening each page. the cycle of the array stopped at http://finishline.com/ all the time. It seems like I get no response. I isolated the situation I just cant get info from that link at all.

it just stops:

... [info] [phantom] Running suite: 2 steps [debug] [phantom] opening url: http://finishline.com/, HTTP GET [debug] [phantom] Navigation requested: url=http://finishline.com/, type=Other, willNavigate=true, isMainFrame=true [debug] [phantom] Navigation requested: url=http://www.finishline.com/, type=Oth er, willNavigate=true, isMainFrame=true

Here is the Isolation code:

var casper = require("casper").create({
    verbose : true,
    logLevel : "debug",
    pageSettings: {
        userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36",
         customHeaders: {
            "Server": "Apache/2.4.1 (Unix)"
        }
     },
    clientScripts: ["jquery.js"] ,
    viewportSize: {
        width: 1920,
        height: 1080
    }
});

casper.start().thenOpen("http://finishline.com")
.then(function(){
    console.log("title : ", this.getTitle())
})
casper.run();

Why can't I scrape or get a response from finishline?

how to handle if I cant get a response to continue with the code when there is no response?

Answer 1

I ran your code using Slimerjs :

casperjs --engine=slimerjs my_casper_script.js

And it ran without issue:

    [info] [phantom] Starting... 
[info] [phantom] Running suite: 2 steps 
[debug] [phantom] opening url: http://finishline.com/, HTTP GET 
[debug] [phantom] Navigation requested: url=http://finishline.com/, type=Undefined, willNavigate=true, isMainFrame=true 
[debug] [phantom] url changed to "http://www.finishline.com/" 
[debug] [phantom] Navigation requested: url=https://a7474150781.cdn.optimizely.com/client_storage/a7474150781.html, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Navigation requested: url=https://a7474150781.cdn.optimizely.com/client_storage/a7474150781.html, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Navigation requested: url=https://cdns.us1.gigya.com/gs/webSdk/Api.aspx?apiKey=3_wvYmaqsebd4zKyXKtXex5iT6qVdAQNm8T5Vjh1LhGavP4EApJp4T5CcdKuXozmAe#origin=http://www.finishline.com/&hasGmid=false, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Navigation requested: url=https://cdns.us1.gigya.com/gs/webSdk/Api.aspx?apiKey=3_wvYmaqsebd4zKyXKtXex5iT6qVdAQNm8T5Vjh1LhGavP4EApJp4T5CcdKuXozmAe#origin=http://www.finishline.com/&hasGmid=false, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Navigation requested: url=https://4978775.fls.doubleclick.net/activityi;src=4978775;type=aa;cat=links00;dc_lat=;dc_rdid=;tag_for_child_directed_treatment=;&ord=0.17442537640363165, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Navigation requested: url=https://4978775.fls.doubleclick.net/activityi;src=4978775;type=aa;cat=links00;dc_lat=;dc_rdid=;tag_for_child_directed_treatment=;&ord=0.17442537640363165, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Navigation requested: url=http://login.dotomi.com/ucm/UCMController?dtm_com=28&dtm_fid=101&dtm_cid=61247&dtm_cmagic=963c2b&dtm_format=5&cli_promo_id=1&dtm_user_id=o867995714&dtmc_ref=&dtmc_loc=http%3A//www.finishline.com/&dtm_user_token=, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Navigation requested: url=http://login.dotomi.com/ucm/UCMController?dtm_com=28&dtm_fid=101&dtm_cid=61247&dtm_cmagic=963c2b&dtm_format=5&cli_promo_id=1&dtm_user_id=o867995714&dtmc_ref=&dtmc_loc=http%3A//www.finishline.com/&dtm_user_token=, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Navigation requested: url=https://20725988p.rfihub.com/ca.html?rfiidc=1048283195230333342&rfiaid=8c49df89b5394c92b2630ec07269f9c3&ver=9&rb=24949&ca=20725988&_o=24949&_t=20725988&pe=https%3A%2F%2F4978775.fls.doubleclick.net%2Factivityi%3Bdc_pre%3DCPyZuc2nuNQCFXah7Qod9IsPqg%3Bsrc%3D4978775%3Btype%3Daa%3Bcat%3Dlinks00%3Bdc_lat%3D%3Bdc_rdid%3D%3Btag_for_child_directed_treatment%3D%3B%26ord%3D0.17442537640363165&pf=http%3A%2F%2Fwww.finishline.com%2F&ra=82836050341159, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Navigation requested: url=https://20725988p.rfihub.com/ca.html?rfiidc=1048283195230333342&rfiaid=8c49df89b5394c92b2630ec07269f9c3&ver=9&rb=24949&ca=20725988&_o=24949&_t=20725988&pe=https%3A%2F%2F4978775.fls.doubleclick.net%2Factivityi%3Bdc_pre%3DCPyZuc2nuNQCFXah7Qod9IsPqg%3Bsrc%3D4978775%3Btype%3Daa%3Bcat%3Dlinks00%3Bdc_lat%3D%3Bdc_rdid%3D%3Btag_for_child_directed_treatment%3D%3B%26ord%3D0.17442537640363165&pf=http%3A%2F%2Fwww.finishline.com%2F&ra=82836050341159, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Navigation requested: url=http://dis.us.criteo.com/dis/dis.aspx?p=3616&cb=23181624377&ref=&sc_r=1280x1024&sc_d=24, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Navigation requested: url=http://dis.us.criteo.com/dis/dis.aspx?p=3616&cb=23181624377&ref=&sc_r=1280x1024&sc_d=24, type=Undefined, willNavigate=true, isMainFrame=false 
[debug] [phantom] Automatically injected jquery.js client side 
[debug] [phantom] Successfully injected Casper client-side utilities 
[info] [phantom] Step anonymous 2/2 http://www.finishline.com/ (HTTP 301) 
title :  Finish Line: Shoes, Sneakers & Athletic Gear 
[info] [phantom] Step anonymous 2/2: done in 7686ms. 
[info] [phantom] Done 2 steps in 7694ms 

when I ran your code using phantomjs as the engine I got the same result as you. The instructions on how to install slimerjs are here. But if you have npm installed you can install it with this command.

npm install slimerjs -g

hope this is of some assistance.