Reputation: 11
this is my first ever post on Stack Overflow.
I am trying to create a C# console application that handles web requests which essentially consists of three steps:
HttpClient
).I've read about the Producer/Consumer pattern using channels which really comes with a lot of interesting options (limit the channel size, enable multithreading, etc.). I have to run this script on an old computer with few RAM, so I want to prevent any type of memory overflow (such as lists or queues that are written faster than executed).
My design idea was to make the input file reader a single producer, which feeds data into the channel. The consumers (one for each HttpClient
) then takes the data and processes it through the API. But I would need an additional consumer (writer to output file).
I don't think this is what Producer/Consumer pattern is for. Is there another way to chain these types of dataflow? I've also thought about adding a simple BlockingCollection
or ConcurrentQeue
at the end, but my guess is that they would not communicate with the producers and don't tell them to await a certain size.
Thank you.
Upvotes: 1
Views: 117