CODEWITHSUNDEEP
pythonhtml
dppham1
dppham1

Reputation: 119

Grabbing the text between tags using BeautifulSoup

I'm trying to grab every piece of individual text between every tag (that is in my list) in a .txt file using beautiful soup and store them into a dictionary. This code works but is terribly slow if I run big files, so is there another way I could go about making this code faster?

from bs4 import BeautifulSoup

words_dict = dict()

# these are all of the tags in the file I'm looking for
tags_list = ['title', 'h1', 'h2', 'h3', 'b', 'strong']

def grab_file_content(file : str):
    with open(file, encoding = "utf-8") as file_object:
        # entire content of the file with tags
        content = BeautifulSoup(file_object, 'html.parser')

        # if the content has content within the <body> tags...
        if content.body:
            for tag in tags_list:
                for tags in content.find_all(tag):
                    text_list = tags.get_text().strip().split(" ")
                    for words in text_list:
                        if words in words_dict:
                            words_dict[words] += 1
                        else:
                            words_dict[words] = 1

       else:
            print('no body')

Upvotes: 2

Views: 240

Answers (1)

DYZ
DYZ

Reputation: 57033

The following code is functionally equivalent to your code:

from collections import Counter    
from itertools import chain

words_dict = Counter() # An empty counter further used as an accumulator

# Probably a loop
# Create the soup here, as in your original code
content = BeautifulSoup(file_object, 'html.parser')
words_dict += Counter(chain.from_iterable(tag.string.split()
                      for tag in content.find_all(tags_list) if tag.string))

Upvotes: 1

Related Questions