CODEWITHSUNDEEP
javaandroidweb-scrapingjsoup
snazzy
snazzy

Reputation: 1

JSoup can't load a website fast enough, but only for me

I am making an android app that scrapes a certain website content and puts it into a list. Lately, I've had an error show up where the Document ( JScoup class ) didn't contain certain elements all of a sudden. I printed the entire document and this is what I got:

<html>
 <head>
    <script type="text/javascript"><!--
    function leastFactor(n) {
    if (isNaN(n) || !isFinite(n)) return NaN;
    if (typeof phantom !== 'undefined') return 'phantom';
    if (typeof module !== 'undefined' && module.exports) return 'node';
    if (n==0) return 0;
    if (n%1 || n*n<2) return 1;
    if (n%2==0) return 2;
    if (n%3==0) return 3;
    if (n%5==0) return 5;
    var m=Math.sqrt(n);
    for (var i=7;i<=m;i+=30) {
        if (n%i==0)      return i;
        if (n%(i+4)==0)  return i+4;
        if (n%(i+6)==0)  return i+6;
        if (n%(i+10)==0) return i+10;
        if (n%(i+12)==0) return i+12;
        if (n%(i+16)==0) return i+16;
        if (n%(i+22)==0) return i+22;
        if (n%(i+24)==0) return i+24;
    }
    return n;
    }
    function go() {
     var p=2456313912482; var s=1782578125; var n;
    if ((s >> 3) & 1)/* 120886108*
    */p+=293087928*/* 120886108*
    */4;/*
    *13;
    */else 
    p-=/* 120886108*
    */202819457*/*
    *13;
    */4;/*
    *13;
    */if ((s >> 1) & 1)
    p+=296193202*/*
    else p-=
    */2;/*
    else p-=
    */else /* 120886108*
    */p-=812964012*2;/*
    p+= */if ((s >> 0) & 1) p+=/*
    p+= */548319846* 3;/*
    p+= */else /*
    p+= */p-=/*
    *13;
    */1210867650*
    1;/* 120886108*
    */if ((s >> 8) & 1) p+=
    151523417*  11;
    else /*
    *13;
    */p-= 212850085*    9;/* 120886108*
    */if ((s >> 4) & 1) p+=/*
    else p-=
    */81741529*
    7;/*
    *13;
    */else /*
    *13;
    */p-=/*
    p+= */18587493*
    5;/*
    else p-=
    */ p-=4349934143;
     n=leastFactor(p);
    { document.cookie="duplexdataiad="+n+"*"+p/n+":"+s+":2100623372:1;path=/;";
      document.location.reload(true); }
    }
//--></script>
</head> 
<body onload="go()">
   Loading...  
</body>

Apparently, <body> is still loading when JSoup scrapes it. What's weird about this is that it only happens sometimes, and when it happens, it only happens to me and only on my internet connection (Ethernet, wifi, neither works), but as soon as I turn on mobile data on my phone and I try to test the app, it suddenly works perfectly. My friend, who is also developing this app with me, had this issue once, a while back, but after that time never. Does anyone know how to fix this?

Upvotes: 0

Views: 258

Answers (1)

Rob Evans
Rob Evans

Reputation: 2874

It seems likely some sort of anti-bot mechanism has detected you and is serving you up bad pages. Hence why it works from one internet connection fine but not the other. I suspect you're IP/some other bit of data/fingerprint has been logged somewhere and being treated as a bot. Really all you can do is slow down your scraping so you're not treated as a bot, OR flip between connections to try and evade detection.

If you try from your browser on what is the 'bad' internet connection, does it work?

Upvotes: 1

Related Questions