CODEWITHSUNDEEP
c#multithreadingasynchronousbackgroundworkerlarge-files
Sam Reina
Sam Reina

Reputation: 305

multithread read and process large text files

I have 10 lists of over 100Mb each with emails and I wanna process them using multithreads as fast as possible and without loading them into memory (something like reading line by line or reading small blocks)

I have created a function which is removing invalid ones based on a regex and another one which is organizing them based on each domain to other lists.

I managed to do it using one thread with: while (reader.Peek() != -1) but it takes too damn long.

How can I use multithreads (around 100 - 200) and maybe a backgroundworker or something to be able to use the form while processing the lists in parallel?

I'm new to csharp :P

Upvotes: 1

Views: 4202

Answers (4)

Foo42
Foo42

Reputation: 3144

I would be inclined to use parallel linq (plinq).

Something along the lines of:

Lists.AsParallel() .SelectMany(list => list) .Where(MyItemFileringFunction) .GroupBy(DomainExtractionFunction)

AsParallel tells linq it can do this in parallel (which will mean the ordering of everything following will not be maintained)

SelectMany takes your individual lists and unrolls them such that all all items from all lists are effectivly in a single Enumerable

Where filers the items using your predicate function

GroupBy collects them by key, where DomainExtractionFunction is a function which gets a key (the domain name in your case) from the items (ie, the email)

Upvotes: 0

Jerry Coffin
Jerry Coffin

Reputation: 490168

Unless the data is on multiple physical discs, chances are that any more than a few threads will slow down, rather than speed up, the process.

What'll happen is that rather than reading consecutive data (pretty fast), you'll end up seeking to one place to read data for one thread, then seeking to somewhere else to read data for another thread, and so on. Seeking is relatively slow, so it ends up slower -- often quite a lot slower.

About the best you can do is dedicate one thread to reading data from each physical disc, then another to process the data -- but unless your processing is quite complex, or you have a lot of fast hard drives, one thread for processing may be entirely adequate.

Upvotes: 5

Justin Pihony
Justin Pihony

Reputation: 67075

You will want to take a look at the Task Parallel Library (TPL).

This library is made for parallel work, in fact. It will perform your action on the Threadpool in whatever is the most efficient fashion (typically). The only thing that I would caution is that if you run 100-200 threads at one time, then you possibly run into having to deal with context switching. That is, unless you have 100-200 processors. A good rule of thumb is to only run as many tasks in parallel as you have processors.

Some other good resources to review how to use the TPL:

Why and how to use the TPL How to start a task.

Upvotes: 1

Uday0119
Uday0119

Reputation: 790

There are multiple approaches to it:

1.) You can create threads explicitly like Thread t = new Thread(), but this approach is expensive on creating and managing a thread.
2.) You can use .net ThreadPool and pass your executing function's address to QueueUserWorkItem static method of ThreadPool Class. This approach needs some manual code management and synchronization primitives.
3.) You can create an array of System.Threading.Tasks.Task each processing a list which are executed parallely using all your available processors on the machine and pass that array to task.WaitAll(Task[]) to wait for their completion. This approach is related to Task Parallelism and you can find detailed information on MSDN

Task[] tasks = null;
for(int i = 0 ; i < 10; i++)
{
     //automatically create an async task and execute it using ThreadPool's thread
     tasks[i] = Task.StartNew([address of function/lambda expression]);
}

try
{
    //Wait for all task to complete
    Task.WaitAll(tasks);
}
catch (AggregateException ae)
{
    //handle aggregate exception here
    //it will be raised if one or more task throws exception and all the exceptions from defaulting task get accumulated in this exception object
}

//continue your processing further

Upvotes: 2

Related Questions