CODEWITHSUNDEEP
pythonbeautifulsouphtml-parsing
alphageek
alphageek

Reputation: 770

Making BeautifulSoup ignore contents inside script tags

I have been trying to get BeautifulSoup (3.1.0.1)to parse a html page that has a lot of javascript that generates html inside tags. One example fragment looks like this :

<html><head><body><div>
<script type='text/javascript'>

if(ii > 0) {
html += '<span id="hoverMenuPosSepId" class="hoverMenuPosSep">|</span>'
}
html += 
'<div class="hoverMenuPos" id="hoverMenuPosId" onMouseOver=\"menuOver_3821();\" ' +
'onMouseOut=\"menuOut_3821();\">';
if (children[ii].uri == location.pathname) {
html += '<a class="hiHover" href="' +  children[ii].uri + '" ' + onClick + '>';
} else {
html += '<a class="hover" href="' +  children[ii].uri + '" ' + onClick + '>';
}
html += children[ii].name + '</a></div>';
}
}          
hp = document.getElementById("hoverpopup_3821");
hp.style.top = (parseInt(hoveritem.offsetTop) + parseInt(hoveritem.offsetHeight)) + "px";
hp.style.visibility = "Visible";
hp.innerHTML = html;
}
return false;
}
function menuOut_3821() {
timeOn_3821 =  setTimeout("showSelected_3821()",  1000)             
}
var timeOn_3821 = null;
function menuOver_3821() {
clearTimeout(timeOn_3821)
}   
function showSelected_3821() {
showChildrenMenu_3821( 
document.getElementById("flatMenuItemAnchor" + selectedPageId), selectedPageId);
}
</script>
</body>
</html>

BeautifulSoup doesn't seem to be able to deal with this and is complaning about "malformed start tag" around the onMouseOver=**\"**menuOver_3821();\". It seems to try parsing the xml that is generated by javascript inside the script block ?!?

Any ideas how to make BeautifulSoup ignores the script tags content ?

I have seen other suggestion of using lxml but can't since it has to run on Google AppEngine.

Upvotes: 1

Views: 1890

Answers (3)

ddaa
ddaa

Reputation: 54544

That would work, but the point of BeautifulSoup is parsing whatever tag soup you throw at it, even if it's horribly ill-formed.

Upvotes: 0

alphageek
alphageek

Reputation: 770

Reverting to BeautifulSoup 3.0.7a solved this issue and many other html oddities that 3.1.0.1 has choked on.

Upvotes: 1

blwy10
blwy10

Reputation: 4892

I've faced this kind of problem before, and what I normally do is replace every occurrence of <script with <!-- and </script> with -->. That way, all the <script></script> tags are commented out.

Upvotes: 0

Related Questions