CODEWITHSUNDEEP
pythonhtmltextbeautifulsoup
Jkiefn1
Jkiefn1

Reputation: 143

How to parse HTML file in .TXT format (un-tabbed) in Python?

I have encountered a problem in my programming that has me stumped.

I'm trying to access data stored in a wealth of old HTML-formatted-saved-as-text files. However, when saving the HTML code lost its indentations, tabs, hierarchy, whatever you wish to call it. An example of this can be found below.

......

<tr class="ro">
<td class="pl " style="border-bottom: 0px;" valign="top"><a class="a" href="javascript:void(0);" onclick="top.Show.showAR( this, 'defref_us-gaap_RevenueFromContractWithCustomerExcludingAssessedTax', window );">Net sales</a></td>
<td class="nump">$ 123,897<span></span>
</td>
<td class="nump">$ 122,136<span></span>
</td>
<td class="nump">$ 372,586<span></span>
</td>
<td class="nump">$ 360,611<span></span>
</td>
</tr>
<tr class="re">
<td class="pl " style="border-bottom: 0px;" valign="top"><a class="a" href="javascript:void(0);" onclick="top.Show.showAR( this, 'defref_us-gaap_OtherIncome', window );">Membership and other income</a></td>
<td class="nump">997<span></span>
</td>
<td class="nump">1,043<span></span>
</td>
<td class="nump">3,026<span></span>
</td>
<td class="nump">3,465<span></span>
</td>
</tr>
<tr class="rou">
<td class="pl " style="border-bottom: 0px;" valign="top"><a class="a" href="javascript:void(0);" onclick="top.Show.showAR( this, 'defref_us-gaap_Revenues', window );">Total revenues</a></td>
<td class="nump">124,894<span></span>
</td>
<td class="nump">123,179<span></span>
</td>
<td class="nump">375,612<span></span>
</td>
<td class="nump">364,076<span></span>
</td>
</tr>

I typically would employ Beautiful Soup here and get to work parsing the data that way, but I've not found a good workflow since technically there is no hierarchy here; I can't tell BS to look within something other than the document itself-which is huge and might be way too time consuming (see next statement).

I also need to find a thorough solution and not a quick-fix because I have hundreds, if not thousands, of these same HTML-to-text files to parse.

So my question here is, if I want to return, in all the files, the first number for "Membership and other Income" (997 in this case), how could I go about doing that?

Two samples files can be found here:

(https://www.sec.gov/Archives/edgar/data/1800/0001104659-18-065076.txt) (https://www.sec.gov/Archives/edgar/data/1084869/0001437749-18-020205.txt)

EDIT - 4/16

Thanks for the replies everyone! I've written some code that returns the tags I'm looking for.

import requests
from bs4 import BeautifulSoup

data = requests.get('https://www.sec.gov/Archives/edgar/data/320193/0000320193-18-000070.txt')

# load the data
soup = BeautifulSoup(data.text, 'html.parser')

# get the data
for tr in soup.find_all('tr', {'class':['rou','ro','re','reu']}):
    db = [td.text.strip() for td in tr.find_all('td')]
    print(db)

The problem is there are a TON of returns and most contain nothing of use. Is there a way to filter based on these tags' grandparent? I've tried the same approach as above using head, title, body, etc. but I can't quite get BS to identify the FILENAME..

<DOCUMENT>
<TYPE>XML
<SEQUENCE>14
**<FILENAME>R2.htm**
<DESCRIPTION>IDEA: XBRL DOCUMENT
<TEXT>
<html>
<head>
<title></title>
.....removed for brevity
</head>
<body>
.....removed for brevity
<td class="text">&#160;<span></span>
</td>
.....removed for brevity
</tr>

Upvotes: 0

Views: 120

Answers (1)

James Lane
James Lane

Reputation: 332

Just so you are aware, HTML does not care about indentation. If you really wanted to, it could all be on the same line with no spaces in between. A HTML parser will just look at the structure of the tags.

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, 'html.parser')
soup.find_all['<tag you are looking for>'][0]

Upvotes: 1

Related Questions