CODEWITHSUNDEEP
pythonpython-2.7beautifulsouphtml-parsing
Slavi
Slavi

Reputation: 130

BeautifulSoup not finding all

I've got the following source code trying to parse a web page however, it appears that it does not find all the instances wherethe class "row dataraekker" appears in

cvr = 45963128
url = 'https://datacvr.virk.dk/data/visenhed?enhedstype=virksomhed&id=%s&soeg=%s' % (str(cvr), str(cvr))

rObject = requests.get(url)
html = rObject.content
soup = BeautifulSoup(html, 'html.parser')
registerHistoryTab = soup.find('div', class_="accordion ", id="accordion-Historisk")
dataRows = registerHistoryTab.find_all('div', class_='row dataraekker')
print len(dataRows)

registerHistoryTab holds 2 items with the following HTML, where multiple div's appear "out of nowhere" as that's not the case in the page's source code

<div class="accordion " data-pdf-class="accordion hide accordion-Historisk" id="accordion-Historisk">
<div class="accordion-group accordion-wrapper">
<div class="accordion-heading">
<div>
<a class="accordion-toggle collapsed" data-parent="#accordion" data-toggle="collapse" href="#collapse_-Historisk">
<h1>Registreringshistorik<div class="text-left help_element help_acc Help-Historisk" data-pdf-class="hide"></div></h1>
</a>
</div>
<!--pdf-linje-->
</div>
<div class="accordion-body collapse" id="collapse_-Historisk">
<div class="accordion-inner" data-pdf-class="sektion">
<div class="aktive-registreringstidende">
<div class="row dataraekker">
<div class="col-sm-12" data-pdf-class="column12">
<b>04.06.2015  Ændring i personkreds</b><br>
<b>CVR-nummer:</b><p>45963128.</p><b>NAVN:</b><p>UNILEVER DANMARK A/S.</p>Adresse: Ørestads Boulevard 73, 2300 København S.<br/>Kommune: København.<br/><b>Bestyrelse:<br/></b><h2>Fratrådte:</h2><p>Jens Christian Voldmester, den 01.06.2015.</p><b>Direktion:<br/></b><h2>Fratrådte:</h2><p>Jens Christian Voldmester, (adm. dir), den 01.06.2015.</p><h2>Tiltrådte:</h2><p>Henrico Drent, (adm. dir), Burgemeester Vogelslaan 63, 5062 KN, Oisterwijk, Holland, den 01.06.2015.</p><hr> </hr></br></div>
</div><div class="row dataraekker">
<div class="col-sm-12" data-pdf-class="column12">
<b>06.03.2015  Øvrige ændringer, Ændring i personkreds</b><br>
<p><b>CVR-nummer: </b>45963128</p><p><b>Navn og adresse: </b></p></br>UNILEVER  DANMARK A/S</div></div></div></div></div></div></div>

The issue appears at the find method because registerHistoryTab is not as it is when viewing the web page

Any help appreciated

Upvotes: 1

Views: 3915

Answers (1)

alecxe
alecxe

Reputation: 474131

The issue appears at the find method because registerHistoryTab is not as it is when viewing the web page

Never expect your HTML returned by requests be the same as you see in the browser. When you deal with HTML parsing, work with what you've got inside the response and what you see in the browser.

Note that in this case, just switching the parser from html.parser to lxml solves the problem:

soup = BeautifulSoup(html, 'lxml')

Now I see 64 printed instead of 2.

Note that this requires lxml to be installed: pip install --upgrade lxml.

Also see:

Upvotes: 3

Related Questions