CODEWITHSUNDEEP
c#multithreading
ShirazITCo
ShirazITCo

Reputation: 1041

C# read text file lines multi thread

I want to write a fast multi thread program using c# that read a file.

so the file must be split into some parts and each part process in different thread. for ex:

Line1
Line2
Line3
Line4

must split to 4 lines like this:

Line1 => thread 1
Line2 => thread 2
Line3 => thread 3
Line4 = > thread 4

i used the StreamReader.readLine() but it cant read specify line.

Comment: its necessary to speedup the program so i want to read file in separate threads.

Upvotes: 0

Views: 3676

Answers (6)

Wojteq
Wojteq

Reputation: 1193

AFAIK .NET doesn't support parallel stream reading. If you want to process every line you may use File.ReadAllLines. It returns an array of strings. Then use you can use PLINQ.

var result = File.ReadAllLine("path")
   .AsParallel()
   .Select(s => DoSthWithString(s))
   .ToList();

Upvotes: 3

Denis
Denis

Reputation: 12087

I would split the file before hand. Say the file is 1000 lines. Split it into 10 files of 100 lines. Have a thread process each file.

Upvotes: 1

Lasse V. Karlsen
Lasse V. Karlsen

Reputation: 391724

You're not going to be able to speed up the actual reading because you're going to have tremendous locking issues keeping everything straight.

Since a text file is an unstructured file, ie. each line can be of different length, you have no choice but to read each line after the other, one by one.

Now, what you can do is process those lines on different threads, but the actual reading, keep that to one thread.

But, before you do that, are you sure you even have to do this? Is this a bottleneck? If not, fix the bottleneck first and see how far you get.

Upvotes: 2

Jon Skeet
Jon Skeet

Reputation: 1503924

You should read the file in a single thread - but then spawn the processing of each line to a different thread, e.g. by adding it to a producer/consumer queue.

Even if you could seek to a specific line in a text file (which in general you can't) you really don't want the disk thrashing around - that'll only slow things down. The fastest way to get the data off the disk is to read it sequentially. By all means defer everything about handling the line beyond "decoding the binary data to text" to other threads, but you really don't want the IO to be in multiple threads.

Upvotes: 5

Bradley Uffner
Bradley Uffner

Reputation: 17001

Your StreamReader is connected to a stream class. Using the stream class you can .Seek to a particular byte location.

Like others have said, this probably isn't a good idea, but it can be done.

Upvotes: 1

Steve
Steve

Reputation: 31662

Unless you're using fixed-length lines, this isn't possible.

Why? Because in order to determine where the "lines" split, you need to find the newline characters... which means you need to read the file first.

Now, if you simply want to perform some extra "processing" after you read in each line - that is possible and relatively straight-forward using a ThreadPool.

Upvotes: 5

Related Questions