CODEWITHSUNDEEP
c#out-of-memory
James D
James D

Reputation: 351

c# Huge memory build-up with HttpWebRequest

Hello I am having problems with httpwebrequest, huge memory buildups when running a rather large request based program. The only real place the build up can be happening is reading the stream back after sending the request, how would I combat this or better yet set a limit to this sort of thing?

I also have various (10-15) if statements looking for various things in the body of the response but about to change them to a switch statement, although I doubt they're causing such high memory buildups I thought it'd be best to mention them. My request looks like so:

HttpWebResponse response = null;

            try
            {
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);


                request.Headers.Set(HttpRequestHeader.CacheControl, "max-age=0");
                request.Headers.Add("Upgrade-Insecure-Requests", @"1");
                request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36 OPR/63.0.3368.88";


                response = (HttpWebResponse)request.GetResponse();

                string _responseData = new StreamReader(response.GetResponseStream()).ReadToEnd();

            catch (WebException e)
            {

            }
            catch (UriFormatException p)
            {


            }

As you can see my stream is read back via _responseData which I then use .contains to see if what I'm looking for is there.

Upvotes: 0

Views: 759

Answers (1)

scopolamin
scopolamin

Reputation: 574

  • Set HttpWebRequest.AllowWriteStreamBuffering to false.
  • Dispose HttpWebResponse, Stream and StreamReader.
            try
            {
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
                request.AllowWriteStreamBuffering = false;

                request.Headers.Set(HttpRequestHeader.CacheControl, "max-age=0");
                request.Headers.Add("Upgrade-Insecure-Requests", @"1");
                request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36 OPR/63.0.3368.88";

                string _responseData;

                using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
                {
                    using (Stream stream = response.GetResponseStream())
                    {
                        using (StreamReader sr = new StreamReader(stream, Encoding.UTF8))
                        {
                            _responseData = sr.ReadToEnd();
                        }
                    }
                }
            }
            catch (WebException e)
            {

            }
            catch (UriFormatException p)
            {

            }

Upvotes: 3

Related Questions