CODEWITHSUNDEEP
pythonrequestbeautifulsoup
김지영
김지영

Reputation: 359

beautifulsoup and request.post

I practice scraping one site. I got some mysterious situation.

import requests
from bs4 import BeautifulSoup
import json

class n_auction(object):
    def __init__(self):
        self.search_request = {
                'lawsup':0,
                'lesson':0,
                'next_biddate1':'',
                'next_biddate2':'',
                'state':91,
                'b_count1':0,
                'b_count2':0,
                'b_area1':'',
                'b_area2':'',
                'special':0,
                'e_area1':'',
                'e_area2':'',
                'si':11,
                'gu':0,
                'dong':0,
                'apt_no':0,
                'order':'',
                'start':60,
                'total_record_val':850,
                'detail_search':'',
                'detail_class':'',
                'recieveCode':'',}

        self.headers = {'User-Agent':'Mozilla/5.0',
              'Referer':'http://goodauction.land.naver.com/auction/ca_list.php'}

    def scrape(self, max_pages):

        addr = []

        pageno = 0
        self.search_request['start'] = pageno
        while pageno < max_pages:
            payload = json.dumps(self.search_request)
            r = requests.post('http://goodauction.land.naver.com/auction/ax_list.php', data=payload ,headers=self.headers)
            print(r.text)


            s = BeautifulSoup(r.text)
            print(s)

if __name__ == '__main__':
    scraper = n_auction()
    scraper.scrape(30)

when I print(r.text), I got full text.like below picture. enter image description here

But after passing through beautifulsoup, I lost some values like below picture. enter image description here

It's very embarrassing. Help me~~

Upvotes: 3

Views: 3389

Answers (1)

user2096803
user2096803

Reputation:

Switching the parser from the default, lxml, to html.parser worked for me.

Try: s = BeautifulSoup(r.text, 'html.parser')

Upvotes: 1

Related Questions