CODEWITHSUNDEEP
c#.net
Rob P.
Rob P.

Reputation: 15071

HttpClient / WebClient - Trouble Interacting With Websites via Proxy

I'd like to use an HttpClient to interact with a website. This is in a corporate-type environment, all web access goes through a web proxy. By default, both the HttpClient and WebClient seem to 'just work' with the proxy here - but I've also specified the proxy details in code.

My problem is that some URLs will load correctly, others will not. The following code shows what I mean.

var webc = new WebClient();

var x1 = webc.DownloadString("http://www.google.com");  // Works
var x2 = webc.DownloadString("http://www.google.ie");   // Works
var x3 = webc.DownloadString("http://maps.google.com"); // Works
var x4 = webc.DownloadString("http://maps.google.ie");  // 403 Forbidden exception

I see the same behaviour with the HttpClient, but the code is more verbose. If I fetch the HTML returned in the 403 error it indicates that I have not authenticated and shows my username as empty.

Using Chrome/FF/IE - I can browse to all four of the sample URLs. The proxy doesn't prevent me, or show the same error message.

It seems like the code only fails on sites that have a non 'www' subdomain - when it is a non .com site. As crazy as that sounds.

I've tried running Fiddler locally to see if anything was different between the requests - from what I can see - it looks identical, except for the URL:

GET http://maps.google.ie/ HTTP/1.1 Host: maps.google.ie Proxy-Connection: Keep-Alive

GET http://www.google.com/ HTTP/1.1 Host: www.google.com Proxy-Connection: Keep-Alive

In the 'Auth' tab fiddler shows:

No Proxy-Authorization Header is present.

No Authorization Header is present.

For both. But the .com example works; and the .ie example fails. I tried pulling up the same maps.google.ie URL from within Chrome - which works great and I can see that it has a Proxy-Authorization in it's GET

GET http://maps.google.ie/ HTTP/1.1 Host: maps.google.ie Proxy-Connection: keep-alive Proxy-Authorization: NTLM T3RMTVNTUAAFAAACB4IBogQABAAzAAAACwALACgAAAAGAbFdAAAAD1BBVUxTT01xOTlEU1UTUR==

Can anyone tell me what's going on here? If that Proxy-Authorization is what I need, how do I get the HttpClient/WebClient to include it? I've tried creating a WebProxy and setting the Credentials on it - with the CredentialCache and with supplying the username/pass/domain (and every variation of the domain name I could think of). When I get it 'wrong' - all the sites seem to return 403. But when I get it right - the top 3 work and the 4th doesn't. In Fiddler, I'm never able to see that Proxy-Authorization in any of the requests I make - but it still works for the 3 first three sites.

I'm sure I've missed something, but I'm at a loss. Any help would be much appreciated.

Upvotes: 0

Views: 1627

Answers (3)

Rob P.
Rob P.

Reputation: 15071

Moby Disk and Aron are both correct, in the sense that those are ways of specifying the proxy. But as mentioned in my question, using them didn't help.

For whatever reason, the web proxy required a User-Agent to be set. Once set, everything worked.

_client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0");

Upvotes: 1

Moby Disk
Moby Disk

Reputation: 3851

There are two ways:

var webc = new WebClient();
webc.UseDefaultCredentials = true;
var x4 = webc.DownloadString("http://maps.google.ie");

or, put this in your app.config

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.net>
    <defaultProxy useDefaultCredentials="true" />
  </system.net>
</configuration>

See tomfanning's answer here: Proxy Basic Authentication in C#: HTTP 407 error

I don't understand why "UseDefaultCredentials" does not default to true. If you work in a corporation that uses a proxy, any app that doesn't do this cannot get out of the LAN.

Upvotes: 1

Aron
Aron

Reputation: 15772

var webc = new WebClient
{
    Proxy = new WebProxy
    {
        Credentials = new NetworkCredential(...),
    }
};


var x1 = webc.DownloadString("http://www.google.com");  // Works
var x2 = webc.DownloadString("http://www.google.ie");   // Works
var x3 = webc.DownloadString("http://maps.google.com"); // Works
var x4 = webc.DownloadString("http://maps.google.ie");  // 403 Forbidden exception

Unfortunately .net is really annoying for programmatically setting the Proxy credentials. You expect you should be able to do this all in config, but it doesn't work out of the box. You can only set the Proxy address in config and not the credentials.

Upvotes: 0

Related Questions