CODEWITHSUNDEEP
c#webauthentication
user2879905
user2879905

Reputation: 1

Web site log in for data scraping

I am attempting to web scrape date from my various remote transmitters. I have one brand of transmitter that I can log into with the following c# code:

public static string getSourceCode(string url, string user, string pass)
{
    SecureString pw = new SecureString();
    foreach (char c in pass.ToCharArray()) pw.AppendChar(c);
    NetworkCredential credential = new NetworkCredential(user, pw, url);
    CredentialCache cache = new CredentialCache();
    cache.Add(new Uri(url), "Basic", credential);
    Uri realLink = new Uri(url);
    HttpWebRequest req = (HttpWebRequest)WebRequest.Create(realLink);
    req.Credentials = CredentialCache.DefaultNetworkCredentials;

    HttpWebResponse resp = (HttpWebResponse)req.GetResponse();

    StreamReader sr = new StreamReader(resp.GetResponseStream());
    string sourceCode = sr.ReadToEnd();
    sr.Close();
    resp.Close();
    return sourceCode;
}

The second brand of transmitter (I'm hesitant to put the url out in public) instead of returning a web page requesting username and password returns a box requesting username and password. using the above code just returns an unauthorized error.

Fiddler says the following is sent when I successfully login to the site:

GET http(colon slash slash)lasvegas3abn(*)dyndns(*)tv(PORT)125(slash)measurements(*)htm HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Accept-Language: en-US
User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0; Touch)
Accept-Encoding: gzip, deflate
Host: lasvegas3abn.dyndns.tv:125
Authorization: Basic dXNlcjpsaW5lYXI=
Connection: Keep-Alive
DNT: 1

Any suggestions?

Upvotes: 0

Views: 207

Answers (1)

adrianbanks
adrianbanks

Reputation: 82934

Instead of:

req.Credentials = CredentialCache.DefaultNetworkCredentials;

you can specify a credential that uses a specific username and password:

req.Credentials = new NetworkCredential("username", "password");

This should enable you to get through the login prompt (assuming that you specify the correct username and password).

Upvotes: 2

Related Questions