CODEWITHSUNDEEP
c#performanceparallel-processingtask-parallel-library
Tono Nam
Tono Nam

Reputation: 36070

Dispose objects read from BlockingCollection

I have to parse a lot of files. So right now I open a file for reading parse the content and write the output on a different location. That is basically what I need to do but I will like to speed up the process since I am parsing 14000 files.

I improved my algorithm by spliting the work on multiple threads. So I had 1 thread do 25% of the files the next 25 % and so on.

Anyways I believe I will dramatically increase performance and speed if I have:

  1. Task 1 read files and place the content of the files in BlockingCollection1 (memory)
  2. Task 2 will create multiple threads to parse the content of BlockingCollection1 and place parsed data in BlockingCollection2
  3. Task 3 read content from BlockingCollection2 and writes it to disk.

The problem that I have now is that I get an out of memory exception. I will like to let the garbage collector remove items from BlockingCollection1 if they have been used. also I will like to remove items from BlockingCollection2 if they have been written to disk.

It is very convenient to use a BlockingCollection<T> since one thread can be adding items to that collection and another thread can be processing those items. Before I used a linked list but I leaned in this question: https://stackoverflow.com/a/12519345/637142 how essay and useful a BlockingCollection can be. Anyways how could I solve this problem? Do I have to use a different type of collection such as a stack?

Upvotes: 1

Views: 1426

Answers (1)

Reed Copsey
Reed Copsey

Reputation: 564931

When you create your BlockingCollection<T> instances, you can use the constructor which accepts a bounding capacity. This prevents you from putting too many items into the collection at once, which can help you throttle the number of items "generated" to help match the number of items consumed.

By keeping this throttled, you can potentially prevent yourself from using all of the memory, as you will not process items faster than you write them to disk.

Upvotes: 3

Related Questions