redteal
redteal

Reputation: 11

Is there a c# design pattern for chaining producer/consumer?

this is my first ever post on Stack Overflow.

I am trying to create a C# console application that handles web requests which essentially consists of three steps:

  1. Read input lines from a file (which can have a length of several million lines).
  2. Each line is seperately fed into a Web API where I await the request, do some interpretation of the response and generate x new lines per input. This step is done through a proxy network of a variable number of different proxies (each proxy corresponding to a single HttpClient ).
  3. All x lines per input are written into the same output file.

I've read about the Producer/Consumer pattern using channels which really comes with a lot of interesting options (limit the channel size, enable multithreading, etc.). I have to run this script on an old computer with few RAM, so I want to prevent any type of memory overflow (such as lists or queues that are written faster than executed).

My design idea was to make the input file reader a single producer, which feeds data into the channel. The consumers (one for each HttpClient) then takes the data and processes it through the API. But I would need an additional consumer (writer to output file).

I don't think this is what Producer/Consumer pattern is for. Is there another way to chain these types of dataflow? I've also thought about adding a simple BlockingCollection or ConcurrentQeue at the end, but my guess is that they would not communicate with the producers and don't tell them to await a certain size.

Thank you.

Upvotes: 1

Views: 117

Answers (0)

Related Questions