CODEWITHSUNDEEP
python
elca
elca

Reputation: 63

Python: Remove Duplicates from Text File

I am new to python. I want to remove duplicate word

and except English word i want to delete all other word and blank line.

purely English word only i want to extract.

i have some text file which contain such like following

aaa
bbb
aaa223

aaa
ccc
ddd

kei60:
sj@6999


jack02
jparkj

so after process duplicate i want to get result following

aaa
bbb
ccc
ddd
jparkj

following is what i tried script source.

if anyone help me much appreciate! thanks!

# read a text file, replace multiple words specified in a dictionary
# write the modified text back to a file

import re

def replace_words(text, word_dic):
    """
    take a text and replace words that match a key in a dictionary with
    the associated value, return the changed text
    """
    rc = re.compile('|'.join(map(re.escape, word_dic)))
    def translate(match):
        return word_dic[match.group(0)]
    return rc.sub(translate, text)



def main():
    test_file = "prxtest.txt"
    # read the file
    fin = open(test_file, "r")
    str2 = fin.read()
    fin.close()

    # the dictionary has target_word:replacement_word pairs
    word_dic = {
    '.': '\n',
    '"': '\n',
    '<': '\n',
    '>': '\n',
    '!': '\n',
    "'": '\n',
    '(': '\n',
    ')': '\n',
    '[': '\n',
    ']': '\n',
    '@': '\n',
    '#': '\n',
    '$': '\n',
    '%': '\n',
    '^': '\n',
    "&": '\n',
    '*': '\n',
    '_': '\n',
    '+': '\n',
    '-': '\n',
    '=': '\n',
    '}': '\n',
    '{': '\n',
    '"': '\n',
    ";": '\n',
    ':': '\n',
    '?': '\n',
    ',': '\n',
    '`': '\n',
    '~': '\n',
    '1': '\n',
    '2': '\n',
    '3': '\n',
    '4': '\n',
    "5": '\n',
    '6': '\n',
    '7': '\n',
    '8': '\n',
    '9': '\n',
    '0': '\n',
    ' ': '\n'}

    # call the function and get the changed text
    str3 = replace_words(str2, word_dic)

    # write changed text back out
    fout = open("clean.txt", "w")
    fout.write(str3)
    fout.close()

if __name__ == "__main__":

    main()

Upvotes: 0

Views: 5223

Answers (5)

Maria Zverina
Maria Zverina

Reputation: 11163

Can be done in two lines:

import re

data ="""aaa
bbb
aaa223

aaa
ccc
ddd

kei60:
sj@6999


jack02
jparkj"""

lines = data.splitlines()   # use f.readlines() instead if reading from file

# split the words and only take ones that are all alpha
words = filter(lambda x: re.match('^[^\W\d]+$', x), lines)
# remove duplicates and print out
print '\n'.join(set(words))

Upvotes: 1

jordanm
jordanm

Reputation: 34914

Something like this should work:

import re
found = []
with open(test_file) as fd:
   for line in fd:
      word = line.strip()
      if word:
         if word not in found and re.search(r'^[[:alpha:]]+$', word):
            print word
            found.append(word)

Upvotes: 1

Vidul
Vidul

Reputation: 10528

Some of the strings in the wanted output may serve as interjections, but the others do not seem to be English words. If pure English words are wanted, a slightly more sophisticated approach is suggested:

import nltk
from nltk.corpus import words


tokens = nltk.word_tokenize(open('prxtest.txt').read())
en_words = [x for x in tokens if x.lower() in words.words()]

# en_words now contains purely English words

Upvotes: 0

marshall.ward
marshall.ward

Reputation: 7198

This will capture lines containing only letters:

fin = open(test_file, 'r')
fout = open('clean.txt', 'w')

s = set()
for line in fin:
    if line.rstrip().isalpha():
        if not line in s:
            s.add(line)
            fout.write(line)

fin.close()
fout.close()

Upvotes: 2

darnzen
darnzen

Reputation: 81

I know this is a python question, but what you're asking seems simpler as a *nix script with grep:

cat infile | grep '^[a-zA-Z]+$' > outfile

If you only want unique lines containing only alpha chars:

cat infile | grep '^[a-zA-Z]+$' | sort -u > outfile

I guess in python you could do:

import re
inf = open('infile', 'r')
for line in inf:
   if (re.match('\A[a-zA-A]+\Z', line):
      print line

Upvotes: 0

Related Questions