CODEWITHSUNDEEP
c#.netweb-crawler
Alireza Noori
Alireza Noori

Reputation: 15253

Is it possible to fake Useragent in C#?

I want to create a crawler with C#. The problem is that some websites have disabled black listed crawlers in their robots.txt file, using:

User-agent: *
Disallow: /

Is there a way I could fake my request to show that I'm for instance Googlebot?

Upvotes: 0

Views: 1597

Answers (2)

Marc Gravell
Marc Gravell

Reputation: 1062975

HttpWebRequest has .UserAgent, however - I would simply say: don't.

Of course, your point re robots.txt is rather moot; that is for you to follow. If you write a badly behaved tool that ignores robots.txt regardless of what you claim as your user-agent, then you should expect to be blacklisted fairly quickly.

In particular, trying to impersonate any of the major players is very dubious. Frankly I'd expect most major sites to also check the incoming IP range.

Upvotes: 3

KristoferA
KristoferA

Reputation: 12397

Yes, the HttpWebRequest has a property for user agent. You can set that to anything.

Upvotes: 0

Related Questions