CODEWITHSUNDEEP
pythonpython-3.x
Stew
Stew

Reputation: 41

Python text help large files

this code works perfectly the only problem it doesnt work with Large txt files . 1GB text file . What can I do to fix?

import os

file_1 = open('file1.txt', 'r', encoding='utf8').read().splitlines()
file_2 = open('file2.txt', 'r', encoding='utf8').read().splitlines()
[file_2.remove(l) for l in file_1 if l in file_2]
with open('file2.txt', 'w') as new_file:
    [new_file.write(l + '\n') for l in file_2]

Upvotes: 0

Views: 47

Answers (2)

Guy
Guy

Reputation: 50819

You need to read the files without saving the content in memory. You can do it by using with on the input files

with open(r'C:\Users\Guy.SA\Desktop\fileB.txt', 'r') as file_2, open(r'C:\Users\Guy.SA\Desktop\fileC.txt', 'w') as new_file:
    for line_2 in file_2:
        with open(r'C:\Users\Guy.SA\Desktop\fileA.txt', 'r') as file_1:
            for line_1 in file_1:
                if line_1 == line_2:
                    break
            else:
                new_file.write(line_2)

Upvotes: 1

Mehrdad Pedramfar
Mehrdad Pedramfar

Reputation: 11073

You should use file object for this:

with open('file1.txt', 'r', encoding='utf8') as file_1,
     open('file2.txt', 'r', encoding='utf8') as file_2:

    for line in file1: # or file 2
        # Do what you need to do with reading it line by line

Also note that:

with will close file automatically after.

Upvotes: 0

Related Questions