CODEWITHSUNDEEP
c#asp.netdebuggingproxyhttpwebrequest
eeeeaaii
eeeeaaii

Reputation: 3460

How do I see the raw HTTP request that the HttpWebRequest class sends?

I know you are all going to answer "use a debugging proxy server like Fiddler" but it's not that simple.

Here's my situation: I have some code that runs on a server, in an ASP.NET page code-behind (aspx.cs), which (among other things) establishes a connection to another server, grabs some stuff, and then formats it and returns it to the browser.

The problem is that the other server is doing the wrong thing, and so I want to be able to pass a debugging flag into the page (via the query string, e.g. ?debug=true) so that it will print out the completely raw HTTP request that it is sending to the other server so I can see what the heck is wrong. This code is running in several places so I want to be able to just pass in this flag on dev, staging, or production and just see the request, without having to figure out whether the production servers can talk to some proxy server that exists somewhere, etc.

You would think that it would be easy to do this, right? So I feel like I'm crazy or something but I looked at the reference for HttpWebRequest and its parent class WebRequest and -- nothing. No can do. You would think Microsoft would have thought of this. The closest thing is that you can access the "Headers" collection but when I tried it, it omitted some really important headers like "content length" -- so it must be "lying" to me (I know it's lying, because I know for a fact that the remote server is returning a 200 status -- the request is successful, it's just returning bad/different/wrong data)

Here is the asked-for code example:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://www.whatever.com");
req.Method = ... whatever ...;
... other setup for the request ...
/* At this point we are about to send the request.
   What does the raw HTTP request look like? */
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();

Upvotes: 88

Views: 105008

Answers (9)

Nathan Schubkegel
Nathan Schubkegel

Reputation: 915

This answer does not include the request content. (But it still may help if you need to diagnose connection issues).

Trace logging of the System.Net code in .NET Core can be captured by making a class that inherits from System.Diagnostics.Tracing.EventListener

See https://github.com/dotnet/runtime/issues/64977 for details + original suggestion.

Example:

private class SystemNetEventListener : EventListener
{
  protected override void OnEventSourceCreated(EventSource eventSource)
  {
    // take anything from System.Net
    if (eventSource.Name?.StartsWith("Private.InternalDiagnostics.System.Net.") == true)
    {
      EnableEvents(eventSource, EventLevel.LogAlways);
    }
  }

  protected override void OnEventWritten(EventWrittenEventArgs eventData)
  {
    try
    {
      // produce a string that can be logged
      var sb = new StringBuilder().Append($"{eventData.TimeStamp:HH:mm:ss.fffffff}[{eventData.EventName}] ");
      for (int i = 0; i < eventData.Payload?.Count; i++)
      {
        if (i > 0)
          sb.Append(", ");
        sb.Append(eventData.PayloadNames?[i]).Append(": ").Append(eventData.Payload[i]);
      }

      Console.WriteLine(sb.ToString().Trim());
    }
    catch
    {
      // on failure... well... we are the logger, so there's nobody to tell we failed
    }
  }
}

Usage:

var listener = new SystemNetEventListener(); // start logging
... // do HTTP/Websocket stuff
listener.Dispose(); // stop logging

Upvotes: 0

jhilgeman
jhilgeman

Reputation: 1583

I know this is an old question, but I was in a tough spot where I didn't control the application config file, so I needed an easy way to enable the tracing via code and then easily access the raw request/response data in an event. So I put together this custom class, HttpRawTraceListener, which might be of use to others that are in my position:

https://github.com/jhilgeman/HttpRawTraceListener/blob/master/HttpRawTraceListener.cs

It was designed to be as simple as adding the file to your project, then calling:

System.Diagnostics.HttpRawTraceListener.Initialize();

...to start the trace. From there, requests/responses will be parsed out of the trace messages and then be made available via the System.Diagnostics.HttpRawTraceListener.FinishedCommunication event.

It's probably not 100% perfect for every scenario (e.g. it's not a proxy, so it won't capture web requests from a browser, for example), but it works pretty well for capturing HttpWebRequests requests/responses to web services, and it might be a good starting point if you need something like this.

Upvotes: 4

kamui
kamui

Reputation: 3419

I realise that this is an old question. @feroze's answer says what to do, but does not go into any detail on how to set up System.Net tracing to achieve it.

As this question was the first Google result for my query into the subject, and as we are all busy people, I thought I would save you all from having to hunt down this information.

System.Web is very powerful for debugging HttpWebRequests and can be easily set up using the web.config:

<configuration>
    <system.diagnostics>

        <trace autoflush="true" /> 

        <sources>
            <source name="System.Net" maxdatasize="1024">
                <listeners>
                    <add name="MyTraceFile"/>
                    <add name="MyConsole"/>
                </listeners>
            </source>
        </sources>

        <sharedListeners>
            <add
              name="MyTraceFile"
              type="System.Diagnostics.TextWriterTraceListener"
              initializeData="System.Net.trace.log" />
                <add name="MyConsole" type="System.Diagnostics.ConsoleTraceListener" />
        </sharedListeners>

        <switches>
            <add name="System.Net" value="Verbose" />
        </switches>

    </system.diagnostics>
</configuration>

Adding a simple HttpWebRequest in your code, and running in debugging mode in Visual Studio, the following information will be displayed in the debug console:

System.Net Verbose: 0 : [6596] WebRequest::Create(https://example.com/service.asmx)
System.Net Verbose: 0 : [6596] HttpWebRequest#62063506::HttpWebRequest(https://example.com/service.asmx#11234)
System.Net Information: 0 : [6596] RAS supported: True
System.Net Verbose: 0 : [6596] Exiting HttpWebRequest#11234::HttpWebRequest() 
System.Net Verbose: 0 : [6596] Exiting WebRequest::Create()     -> HttpWebRequest#11234
System.Net Verbose: 0 : [6596] HttpWebRequest#11234 ::GetRequestStream()
System.Net Verbose: 0 : [6596] ServicePoint#11234 ::ServicePoint(example.com:443)
System.Net Information: 0 : [6596] Associating HttpWebRequest#11234with ServicePoint#11234
System.Net Information: 0 : [6596] Associating Connection#11234 with HttpWebRequest#11234 
System.Net Information: 0 : [6596] Connection#11234 - Created connection from x.x.x.x:xx to x.x.x.x:xx.
System.Net Information: 0 : [6596] TlsStream#11234 ::.ctor(host=example.com, #certs=0)
System.Net Information: 0 : [6596] Associating HttpWebRequest#11234 with ConnectStream#11234 
System.Net Verbose: 0 : [6596] Exiting HttpWebRequest#11234 ::GetRequestStream()    -> ConnectStream#11234 
System.Net Verbose: 0 : [6596] ConnectStream#7740977::Write()
System.Net Verbose: 0 : [6596] Data from ConnectStream#11234::Write
System.Net Verbose: 0 : [6596] 00000000 : 3C 73 6F 61 70 3A 45 6E-76 65 6C 6F 70 65 0D 0A : <soap:Envelope..
...etc

I found this especially useful when trying to find out the cause of a webservice client error. It turned out I was missing a header.

Upvotes: 182

Mroczny Arturek
Mroczny Arturek

Reputation: 419

I suggest you to download Telerik Fiddler to capture incoming/outcoming traffic.

Over here is simple example how to do it by that tool:

  1. Be sure that the Capture Traffic is enabled: enter image description here
  2. Open browser and refresh the page, or just send request via HTTP client. enter image description here
  3. After that switch to the Fiddler, you should see your request: enter image description here
  4. At the top try to navigate “Raw” tab. enter image description here
  5. In the below window is your raw request enter image description here

Upvotes: 3

feroze
feroze

Reputation: 7594

You can use System.Net tracing mechanism to see the raw HTTP requests sent on the wire. You can also add your own tracelistener to the process.

Upvotes: 14

eeeeaaii
eeeeaaii

Reputation: 3460

answering my own question here, because I thought of another way to do it. Basically the idea is -- you re-point the HttpWebRequest to a page that logs the incoming raw HTTP Request. In other words set up a custom HTTP handler as per this forum post:

http://forums.asp.net/t/353955.aspx

And then change just the URL in the HttpWebRequest to point to this new endpoint, but keep all other elements of the request the same. Write the result to a file or something and you're golden.

Upvotes: 5

yfeldblum
yfeldblum

Reputation: 65445

You say that you think .NET is lying to you, and the specific example you give is that the header Content-Length is missing from the HTTP response.

But the header Content-Length is not required from an HTTP response. In fact, if the body of the response is in any dynamic, and if its length is not known in advance, then it is highly likely that the Content-Length header will be omitted!

Upvotes: 0

Mike Atlas
Mike Atlas

Reputation: 8231

Another suggestion. Implement your own web proxy, and set your request to use it with WebRequest.Proxy. Then you should be able to extract the traffic from the proxy instance.

Edit: update for links.

Upvotes: 1

Oded
Oded

Reputation: 499352

You can use a network traffic sniffer like wireshark.

This is not a debugging proxy, but will sniff all traffic and let you see the raw requests/responses.

Upvotes: 5

Related Questions