CODEWITHSUNDEEP
pythontxt
Yvar
Yvar

Reputation: 155

Read txt file and remove duplicates

I do open a txt file with the below code and getting the following output: ['MUR,MUR,COP,COP,COP,COP,COP']

# opening the file in read mode
my_file = open("my_txt.txt", "r")

# reading the file
data = my_file.read()

# replacing end splitting the text 
# when newline ('\n') is seen.
data_into_list = data.split("\n")
print(data_into_list)
my_file.close()

How can I remove the duplicates?

Upvotes: 0

Views: 84

Answers (2)

Henro Sutrisno Tanjung
Henro Sutrisno Tanjung

Reputation: 510

use built-in data type python, call : set Set items are unordered, unchangeable, and do not allow duplicate values.

for example :

data = ['MUR,MUR,COP,COP,COP,COP,COP']

>>> data_new = list(data[0].split(','))
>>> data_new
['MUR', 'MUR', 'COP', 'COP', 'COP', 'COP', 'COP']
>>> data_uniq = list(set(data_new))
>>> data_uniq
['MUR', 'COP']
>>>

Upvotes: 2

Nikolay Zakirov
Nikolay Zakirov

Reputation: 1584

use set()

data_into_list = list(set(data.split("\n")))

I suggest you look at this question if you need to preserve order How do I remove duplicates from a list, while preserving order?

data_into_list = list(dict.fromkeys(data.split("\n")))

Upvotes: 2

Related Questions