CODEWITHSUNDEEP
pythonhtml-parsinglxml
Granny Aching
Granny Aching

Reputation: 1435

Using Python, how to extract information from HTML files based on id tag?

I'm trying to create a python script that will extract information from some HTML files. I have no problems with os and glob to get all the necessary files. But the hard part is parsing those files. Here's my code so far:

from lxml import etree
...
parser = etree.HTMLParser(remove_comments=True, recover=True)
tree = etree.parse(os.path.join(path, filename), parser=parser)
...
for item in tree.getiterator():
    id = item.attrib.get('id', None)

    if item.tag == 'title':
        device.name = item.text
    elif id:
        setattr(device, id, item.text)

This code seems to work on some info in the file, such as this one:

<td id="type">Network Camera</td>

but then the HTML files have several lines like this one:

<td colspan="2"><span id="name"></span>:&nbsp;XYZ</td>

I'm not getting anything useful. I inserted print statements, and I can see elements td (with no id and no text) and span (with id, but also no text).

Then there's this one:

<td><table><tr>
    <td><a href="..." id="ipLink"> <span id="ipTxt"></span></a>:&nbsp;
    </td><td>
        1.2.4.3&nbsp;(<span id="staTxt"></span>)
    </td>
</tr></table></td>

... which seems obvious to my human eyes that I should be getting ip=1.2.4.3, but I have no idea how to convince python to extract this.

update:

Complete sample input file:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
    <meta http-equiv="Pragma" content="no-cache">
<title>AXIS M3037</title>
</head>
<body>


<table>
  <tr>
    <td id="type">Network Camera</td>
    <td>|</td>
    <td valign="middle" align="left" width=169 class="menuActive" id="mainMenu" nowrap>

    </td>
    <td><a href="/" id="tLViewTxt"><span id="ti2LViewTxt"></span></a></td>
    <td><a href="/?id=171" id="tSetTxt"><span id="ti2SetTxt"></span></a></td>
    <td colspan="2"><span id="version"></span>:&nbsp;1.23</td>

    <td>
        1.2.1.1&nbsp;(<span id="xyz"></span>)
    </td>
    <td colspan="2">
        <a href="/?id=171" id="dateTimeLink">
            <span id="datTimTxt"></span>
        </a>&nbsp;
        <input type="text" name="CurrentServerDate" value="2018-08-14" disabled>
        &nbsp;&nbsp;&nbsp;
        <input type="text" name="CurrentServerTime" value="11:03:49" disabled>
    </td>

    <td><table><tr>
        <td><a href="..." id="ipLink">
                <span id="ipTxt"></span>
            </a>:&nbsp;
        </td><td>
            1.2.4.3&nbsp;(<span id="staTxt"></span>)
        </td>
    </tr></table></td>
  </tr>
  <tr>
    <td nowrap colspan="2">:&nbsp;
        1
        &nbsp;<span id="videoTxt"></span>&nbsp;&nbsp;
        0
        &nbsp;<span id="audTxt"></span>
        &nbsp;&nbsp;</td>
    <td colspan="2" nowrap>
        <span id="upTimTxt"></span>&nbsp;
        <span id="theuptimevalue">130 days, 3:40</span></td>
  </tr>
</table>
</body>
</html>

Desired extracted information:

'type': 'Network Camera'
'version': '1.23'           (or ': 1.23'  --- I can remove ':')
'xyz': '1.2.1.1'
'staTxt': '1.2.4.3'         (or better: 'ipTxt': '1.2.4.3' )
'videoTxt': '1'
'audTxt': '0'
'theuptimevalue': '130 days, 3:40'

Upvotes: 0

Views: 466

Answers (1)

Jack Fleeting
Jack Fleeting

Reputation: 24930

Well, the following is pretty convoluted and probably brittle, but it does the trick on the html provided:

from lxml.html import fromstring
data = [your html above]

tree = fromstring(data)

for typ in tree.xpath("*//td[@id='type']"):
    print('type',typ.text)
for spa in tree.xpath("*//span[@id='version']/../text()"):
    print('version',spa)
for spa in tree.xpath("*//span[@id='name']/../text()"):
    print(spa.replace(':','').strip(),tree.xpath("*//span[@id='name']/../following-sibling::td/text()")[0].strip())
for spa in tree.xpath("(*//span[@id='staTxt']/..)[2]"):
    print('ipTxt',spa.text.strip())
for spa in tree.xpath("*//span[@id='videoTxt']/.."):
    print('videoTxt',spa.text.replace(':','').strip())  
for spa in tree.xpath("*//span[@id='audTxt']/.."):
    num = "".join(spa.text_content().split())
    print('audTxt2',num[2])
for spa in tree.xpath("*//span[@id='theuptimevalue']"):
    print('theuptimevalue',spa.text.replace(':','').strip())

Output:

type Network Camera
version : 1.23
XYZ 1.2.1.1
ipTxt 1.2.4.3
videoTxt 1
audTxt2 0
theuptimevalue 130 days, 340

You can probably improve on it if you play with it, but is should be a start...

Upvotes: 1

Related Questions