CODEWITHSUNDEEP
javahtmlweb-crawlerjsoup
Victor S
Victor S

Reputation: 4201

Why jsoup-java return null html page via connection?

My code:

        Document document = Jsoup.connect(pageUrl)
            .userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36")
            .get();

        System.out.println("parse document: \n"+document.toString());

to get html page in "https://www.mmbang.com/bang/509/907187", but response is: 

<html>
 <head>
  <script>
    (function(window){
        function createCookie(name,value,days, domain) {
            if(name == 'http_referrer'){
                var exp = new Date();
                exp.setTime(exp.getTime() + 10000);
                document.cookie = name+"="+value+";expires="+exp.toGMTString();
                return;
            }
            if (days) {
                var date = new Date();
                date.setTime(date.getTime()+(days*24*60*60*1000));
                var expires = "; expires="+date.toGMTString()+"; path=/; domain="+domain;
            }
            else var expires = "";
            document.cookie = name+"="+value+expires;
        }
        createCookie("http_referrer", document.referrer, 1, '.mmbang.com');
        createCookie("__9XRwuHhHTjQiC8wXCnCj7wsEBU9x", 'wy', 9000000, '.mmbang.com');
        location.reload();
    })(window);
    </script>
 </head>
 <body></body>
</html>

what's meaning? and how should I do to work it out?

Upvotes: 0

Views: 66

Answers (1)

Luk
Luk

Reputation: 2256

This page requires referrer header in request.

Try using: 

    Document doc = Jsoup.connect(url) //
            .userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") //
            .referrer("www.mmbang.com")
            .get();

Upvotes: 1

Related Questions