CODEWITHSUNDEEP
pythonunicodeencodingutf-8
Emre
Emre

Reputation: 1103

Read from file with mixed unicode characters and replace string (python)

Here is the file content:

H\u00f6gskolan
Högskolan

(Note: \u00f6 is actually ö)

I want to replace all ö characters with o, and save it to a new file. The regular string replace doesn't work as expected.

with open("test.txt", encoding="utf-8") as f:
  content = f.read()
replacedContent = content.replace("ö", "o")
with open("testOutput.txt", "w", encoding="utf-8") as f:
  f.write(replacedContent)

testOutput.txt file content:

H\u00f6gskolan
Hogskolan

I found a similar question but its solution didn't work as well:

import codecs

with open("test.txt", encoding="utf-8") as f:
    content = f.read()
content = codecs.decode(content, "unicode_escape")
replacedContent = content.replace("ö", "o")
with open("testOutput.txt", "w", encoding="utf-8") as f:
    f.write(replacedContent)

testOutput.txt file content:

Hogskolan
Högskolan

Any idea how to achieve this?

Upvotes: 1

Views: 334

Answers (1)

Gevorg Chobanyan
Gevorg Chobanyan

Reputation: 157

I couldn't find any library function that already handles what you are requesting, so I just parse the raw unicode signatures with regex and replace them with their actual values

import re


def convert_mixed(from_file, to_file):
    f = open(from_file, "r")
    content = f.read()
    f.close()
    for x in re.findall(r"\\u([\da-f]{4})", content):
        content = content.replace(f"\\u{x}", chr(int(x, 16)))

    f = open(to_file, "w")
    f.write(content.replace("ö", "o"))


convert_mixed("file.txt", "out.txt")

Upvotes: 1

Related Questions