CODEWITHSUNDEEP
pythonhtmlregexscreen-scrapinghtml-content-extraction
pns
pns

Reputation: 423

Python HTML scraping

It's not really scraping, I'm just trying to find the URLs in a web page where the class has a specific value. For example:

<a class="myClass" href="/url/7df028f508c4685ddf65987a0bd6f22e">

I want to get the href value. Any ideas on how to do this? Maybe regex? Could you post some example code? I'm guessing html scraping libs, such as BeautifulSoup, are a bit of overkill just for this...

Huge thanks!

Upvotes: 3

Views: 3880

Answers (7)

George Godik
George Godik

Reputation: 1716

read Parsing Html The Cthulhu Way https://blog.codinghorror.com/parsing-html-the-cthulhu-way/

Upvotes: 0

ghostdog74
ghostdog74

Reputation: 342799

If your task is just this simple, just use string manipulation (without even regex)

f=open("htmlfile")
for line in f:
    if "<a class" in line and "myClass" in line and "href" in line:
        s = line [ line.index("href") + len('href="') : ]
        print s[:s.index('">')]
f.close()

HTML parsers is not a must for such cases.

Upvotes: 1

pns
pns

Reputation: 423

The thing is I know the structure of the HTML page, and I just want to find that specific kind of links (where class="myclass"). BeautifulSoup anyway?

Upvotes: 0

John Keyes
John Keyes

Reputation: 5604

Regex should not be used to parse HTML. See the first answer to this question for an explanation :)

+1 for BeautifulSoup.

Upvotes: 1

Mark Byers
Mark Byers

Reputation: 838876

Regex would be a bad choice. HTML is not a regular language. How about Beautiful Soup?

Upvotes: 2

Yacoby
Yacoby

Reputation: 55465

Regex is usally a bad idea, try using BeautifulSoup

Quick example:

html = #get html
soup = BeautifulSoup(html)
links = soup.findAll('a', attrs={'class': 'myclass'})
for link in links:
    #process link

Upvotes: 16

Daniel Roseman
Daniel Roseman

Reputation: 599866

Aargh, not regex for parsing HTML!

Luckily in Python we have BeautifulSoup or lxml to do that job for us.

Upvotes: 9

Related Questions