Tag and string mixed find-and-replace using BeautifulSoup in python

Question

How do you use the BeautifulSoup .replace_with() without having something like sharp brackets being converted to > thing after a str() string conversion find-and-replace process?

Python code

from bs4 import BeautifulSoup

with open("../dicttest.txt", "r", encoding="utf-8") as f:
    full_text = f.read()
    parse_1 = BeautifulSoup(full_text, "html.parser")
    for line in parse_1.find_all("grace", "AllExamples"):
        match = str(line).replace("; ", ";
 ")
        line.replace_with(match)
        print(parse_1)

dicttest.txt

all Duden-Oxford Deutsch-Englisch all /al/ Indefinitpron. u. unbest. Zahlw. 1. attr. (ganz, gesamt...) all; in aller Deutlichkeit in all clarity; alle Freude, die sie empfunden hat all the joy she felt; alles Geld, das ich noch habe all the money I have left; aller Eifer nützte ihm nichts all his zeal was to no avail; ich kann diese Leute alle nicht leiden I can't stand any of these people; ich will euch alle nicht mehr sehen I don't want to see any of you again; die Ärzte verdienen alle sehr viel doctors all earn a great deal; alles Geld spendete sie dem Roten Kreuz she donated all her money to the Red Cross; alles Leid der Welt all the suffering in the world; all unser/mein usw. ... all our/my etc. ...; alles andere/Weitere/Übrige everything else; alles Übrige hat sich nicht geändert nothing else has changed; alles Schöne/Neue/Fremde everything or all that is beautiful/new/strange; alles Gute! all the best!; alle Fenster schließen close all the windows; sie gaben alle Waffen ab they handed in all their weapons; wir/ihr/sie alle all of us/you/them; we/you/they all; das sagen sie alle (ugs.) that's what they all say; alle Beteiligten/Anwesenden all those involved/present; trotz aller Vorbehalte werde ich ... in spite of all my reservations I shall ...; alle beide/alle zehn both of them/all ten of them; alle Männer/Frauen/Kinder all men/women/children; alle Mädchen über zwölf Jahre all girls over twelve; alle Mädchen in der Schule all the girls in the school; alle Bewohner der Stadt all the inhabitants of the town; ohne allen Anlass for no reason [at all]; without any reason [at all]; gegen alle Erwartungen contrary to all expectations; alle Jahre wieder every year; alle fünf Minuten/Meter every five minutes/metres; Bücher aller Art books of all kinds; all kinds of books; in aller Eile with all haste; in aller Ruhe in peace and quiet; trotz aller Versuche/Anstrengungen despite all [his/her/their/etc.] attempts/efforts. 2. allein stehend a) (gesamt..., sämtlich) everything; alles geht vorüber everything passes [in time]; alles für die Braut/den Bastler everything for the bride/handicraft enthusiast; das alles all that; ich weiß nicht, was das alles soll I don't know what all that is supposed to mean; das ist alles Unsinn that is all nonsense; von allem etwas verstehen/wissen understand/know a bit about everything; wer alles war od. wer war alles dort who was there?; wen alles habt ihr getroffen? who did you meet?; das sind alles Gauner they're all scoundrels; was gab es dort alles zu sehen? what was there to see?; was es nicht alles gibt! well, would you believe it!; well, I never!; all[es] und jedes everything; (wahllos) anything and everything; trotz allem in spite of or despite everything; sie liebt ihren Hund über alles she loves her dog more than anything else; zu allem fähig sein (fig.) be capable of anything; alles schon mal da gewesen (ugs.) it's all happened before; das kenne ich alles schon I've heard it all before; alles in allem all in all; vor allem above all; alles klar od. in Ordnung (ugs.) everything's fine or (coll.) OK; alles klar? everything all right or (coll.) OK?; dann treffen wir uns um 5⁰⁰ Uhr, alles klar? we'll meet at 5 o'clock then, all right or (coll.) OK?; das ist alles that's all or (coll.) it; ist das alles? is that all or (coll.) it?; nach allem, was man hört/weiß to judge from everything or all one hears/knows; b) (jeder einzelne) everyone; alle miteinander all together; ihr seid/wir sind/sie sind ..., alle miteinander you/we/they are ..., all of you/us/them; alle auf einmal all at once; sprecht nicht alle auf einmal! don't all speak at once; am besten, wir gehen alle auf einmal zum Chef the best thing would be for us all to go and see the boss together; alle, die ... all those who ...; der Kampf aller gegen alle unfettered competition; in allem einverstanden sein agree or be agreed on everything; von allem etwas nehmen take a bit of everything; er ist bei allem, was er tut, sehr genau he is very precise in everything he does; sie ist in allem sehr empfindlich she is very sensitive about everything; c) (Neutr. Sg.: alle Beteiligten) alles mal herhören! (ugs.) listen everybody!; (stärker befehlend) everybody listen!; alles war nach Hause gegangen (ugs.) everyone or everybody had gone home; alles aussteigen! (ugs.) everyone or all out!; (vom Schaffner gesagt) all change! a, A Duden-Oxford Deutsch-Englisch a, A /a:/ das; a/A, a/A a) (Buchstabe) a/A; kleines a small a; großes A capital A; das A und O (fig.) the essential thing/things (Gen. for); von A bis Z (fig. ugs.) from beginning to end; wer A sagt, muss auch B sagen (fig.) if one starts a thing, one must go through with it; b) (Musik) [key of] A

The whole story:

I'm making a HTML based dictionary in python using BeautifulSoup and Regular Expressions. The structure of the dictionary is mainly like this:

Headword | IPA

Article 1
...Article A
......All Examples (say German example with an English explaining)
......German example
......English explaining;
......German example
......English explaining;
......and so forth...
...Article B
......All Examples
......and so forth...

In order to arrange all of them by CSS, I have to assign CSS classes to every element (Articles, examples...) in it. I used to do all of this in pure notepad environment with Regex find-and-replace. Everything works fine except for the fact that I want to process text chunk by chunk, namely I don't want a Regex to affect more than the part I'm working with. Say the element AllExamples, I give them a whole class AllExamples first, then give the German example and English explaining different classes and add s following those semicolons at the end of English explainations. That's not easy, because:

This can't be done with pure notepad environment with a single Regex find-and-replace. In Editpad Pro, I can match the whole AllExample class by a Regex then use a second Regex to replace ; with ; within the matched selection. It's fine if there are few instances to process, but a whole dictionary needs an one-click batch processing.

The reason why I have to match an area first is that there are many equivalent patterns somewhere out of the area which I don't want to touch.

There are exceptions in the structure. Notice the second English explaining with an i tag at the end, that's where my Regex adding follows; fails. So in this case, I have to replace ; with ;. Again, the whole AllExample class should be matched first, because of those equivalent patterns outside the area.

So BeautifulSoup is the solution, I can easily match the area with it and feed a simple .replace() to it. Here the problem is BeautifulSoup treat tags and strings as totally different things. However in my case, the tag and needs to be matched with a ; which is a string.

So I'm going to mix tags and strings together and then do a find-and-replace like in a notepad environment. (I know some of you guys may create a certain complicated function in python to do this, but it seems hard for me.)

Then use the .replace_with() function to give it back to BeautifulSoup like the topic I quoted at the beginning of my post. When I do this, however, all the sharp brackets change to like > in the resulting print. What should I do to solve this issue please?

Related topic here:
Python - Find text using beautifulSoup then replace in original soup variable

Martijn Pieters · Accepted Answer

Your mistake is treating the HTML tags as text here. You serialised the BeatifulSoup object tree to a HTML string, manipulated that string, then told BeatifulSoup about a new text element. Text elements (NavigableText) are not tags, and anything HTML-like will be escaped. You'd have to unserialise the text back to a HTML structure.

The 'proper' way to handle this is to insert a new tag in the right places. Your text replacement shows the rules:

Inside a tag, find any element whose text ends in ;, and which is followed by a tag.

For each such element, insert a right after.

I'd just search for tags inside tags, then filter. Once you found a match, use Tag.insert_after() to add a new tag:

for emphasis in sparse_1.select('grace.AllExamples i'): # must have text that ends in ; if emphasis.string is None or not emphasis.string.endswith(';'): continue # must have a bold tag next next_tag = emphasis.find_next_sibling() if not next_tag or next_tag.name != 'b': continue # match confirmed, insert a break tag emphasis.insert_after(parse_1.new_tag('br'))

You could fold the text check and next_sibling checks into a generator function too, or in a function that's used to check each element in a .find_all() operation, but the above is probably the right level of encapsulation for this problem if there are related replacements you need to make.

In short, don't think of your HTML as a large body of text, but as a directed tree with nodes, where nodes are either tags or text elements. Use BeautifulSoup to navigate that tree and then, when in the right locations, manipulate the tree by adding or removing nodes as needed.

Tag and string mixed find-and-replace using BeautifulSoup in python

Answers (2)

Related Questions