CODEWITHSUNDEEP
pythonregexbeautifulsoup
rakslice
rakslice

Reputation: 8975

When using BeautifulSoup 4's `find_all` with a regex, how do I access regex match capture groups?

I'm using BeautifulSoup 4, and I'm using find_all with a regular expression to find all the links matching a particular pattern.

results = page.find_all(href=re.compile("foo/bar\?baz="))
for result in results:
    ...

However I also want to extract a parameter from the URL.

I can mark the parameter for extraction by putting a capture group on it:

results = page.find_all(href=re.compile("foo/bar\?baz=([^&]+)"))

But if I do this, how do I access the value of the capture group in a particular match?

Upvotes: 3

Views: 980

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195448

Yes, you can. Make helper class with magic methods __call__() and __iter__() and supply instance of this class as a function to BeautifulSoup find_all() function. I used zip() to tie the groups with matched elements:

from bs4 import BeautifulSoup, Tag
import re

data = '''<div>
<a href="link_1">Link 1</a>
<a href="link_2">Link 1</a>
<a href="link_XXX">Link 1</a>
<a href="link_3">Link 1</a>
</div>'''

soup = BeautifulSoup(data, 'lxml')

class my_regex_searcher:
    def __init__(self, regex_string):
        self.__r = re.compile(regex_string)
        self.groups = []

    def __call__(self, what):
        if isinstance(what, Tag):
            what = what.name

        if what:
            g = self.__r.findall(what)
            if g:
                self.groups.append(g)
                return True
        return False

    def __iter__(self):
        yield from self.groups

searcher = my_regex_searcher(r'link_(\d+)')
for l, groups in zip(soup.find_all(href=searcher), searcher):
    print(l)
    print(groups)

searcher = my_regex_searcher(r'(d)(i)(v)')
for l, groups in zip(soup.find_all(searcher), searcher):
    print(l.prettify())
    print(groups)

Prints:

<a href="link_1">Link 1</a>
['1']
<a href="link_2">Link 1</a>
['2']
<a href="link_3">Link 1</a>
['3']
<div>
 <a href="link_1">
  Link 1
 </a>
 <a href="link_2">
  Link 1
 </a>
 <a href="link_XXX">
  Link 1
 </a>
 <a href="link_3">
  Link 1
 </a>
</div>
[('d', 'i', 'v')]

Upvotes: 1

Related Questions