CODEWITHSUNDEEP
vb.nettextout-of-memory
Marc Intes
Marc Intes

Reputation: 737

Opening 90 million lines of text from a .txt file

I am having problems in opening 90 million lines of text from a .txt file and adding them in the array, it is giving me an error and it prevents it from opening. Here is my code:

Dim Contents As String()
Contents = File.ReadAllLines(RichTextBox4.Text)

I have tried 1 million lines of text and it works, but when i tried over 90 million lines it is now giving me an error. Is there any alternative for this? Thanks in advance.

additional info:

These 90 million lines acts as a filter which prevents the user from inputting if the word they input exists in one of the 90 million lines. That is my main goal.

Upvotes: 1

Views: 1961

Answers (2)

Rami
Rami

Reputation: 7290

You can read the huge file in chunks and save each chunk on the disk for later reference.

You can get the underlying file stream and then do something like that:

Dim stream = reader.BaseStream

then use the stream seek method to move the file pointer 

stream.Seek(CHUNK_SIZE, SeekOrigin.[Start])

Upvotes: 1

Allan S. Hansen
Allan S. Hansen

Reputation: 4081

I'd suggest using a steam reader and read a single line each time and then handle that and then read the next line while 'throwing away' the old line.

That way you do not need to read every single line into memory but only the working line.

Upvotes: 4

Related Questions