Calling a web service with asp.net that needs cookies

Question

I'm trying to use the HTML Agility Pack to scrape the list of lawyers at http://www.dlapiper.com/global/people/search.aspx?gLastName=A. The site uses Ajax to populate the names, so I haven't had any luck doing that server side. I noticed the site is calling a web service at http://www.dlapiper.com/FCWSite/DlaPiperWS/Attorneys.asmx?op=FindAttorneys so I'm trying to call the same web service from my application.

I'm using the following code, but it's returning an Internal Server Error 500, which I suspect is because I need a session cookie? (as the site doesn't work if I disable cookies)

Can anyone help?

private static string WebServiceCall()
{

    WebRequest webRequest = WebRequest.Create("http://www.dlapiper.com/FCWSite/DlaPiperWS/Attorneys.asmx?op=FindAttorneys");
    HttpWebRequest httpRequest = (HttpWebRequest)webRequest;
    httpRequest.Method = "POST";
    httpRequest.ContentType = "text/xml; charset=utf-8";
    httpRequest.Headers.Add("SOAPAction: http://tempuri.org/");
    httpRequest.ProtocolVersion = HttpVersion.Version11;
    Stream requestStream = httpRequest.GetRequestStream();
    //Create Stream and Complete Request             
    StreamWriter streamWriter = new StreamWriter(requestStream, Encoding.ASCII);
    string request = "<soap:Envelope xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/'><soap:Body><FindAttorneys xmlns='FCWSite.FCWSite.DlaPiperWS'><scFirstName></scFirstName><scLastName>A</scLastName><scKeyword></scKeyword><scOfficesGUID></scOfficesGUID><scSpokenLanguagesGUID></scSpokenLanguagesGUID><scServicesGUID></scServicesGUID><scRegionGUID></scRegionGUID><scSchoolGUID></scSchoolGUID><scAdmissionGUID></scAdmissionGUID><scLevelGUID></scLevelGUID><strLanguageGUID>7483b893-e478-44a4-8fed-f49aa917d8cf</strLanguageGUID><strCountry>global</strCountry><returnUntranslated>true</returnUntranslated><sortBy>name</sortBy><page>0</page></FindAttorneys></soap:Body></soap:Envelope>";

    streamWriter.Write(request);
    streamWriter.Close();
    //Get the Response    
    HttpWebResponse wr = (HttpWebResponse)httpRequest.GetResponse();
    StreamReader srd = new StreamReader(wr.GetResponseStream());
    string resulXmlFromWebService = srd.ReadToEnd();
    return resulXmlFromWebService;
}

Answer 1

As I stated in my comment.. I added the Attorney.asmx url as a web reference in my project.

When doing so, I let the web reference have the namespace of the reference be 'com.dlapiper.www'.

The following is in vb.net, and represents a standard default page. The class com.dlapiper.www.Attorneys represents the context of the connection.

Note that Attorneys.FindAttorney returns a string value..

Also, you can see from my buffer and array.convertAll attempt in the code, that I was trying different encodings.. if you attempt to invoke FindAttorney without adding the cookie container and the following cookies.. you'll incorrectly get a SOAP exception stating the localizationGUID was not in a proper input.

I left this somewhat irrelevant portion in the code because I'm a fan of contravariant programming.

the truth is, that parameter can't be empty, but isn't actually used by function.. the cookie name:value is.

VB.NET

Public Class _Default
    Inherits System.Web.UI.Page

    Dim AttorneyList As com.dlapiper.www.Attorneys = New com.dlapiper.www.Attorneys()
    Dim attys As String

    Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
        Dim buffer As Byte()
        buffer = Array.ConvertAll(Of Char, Byte)("7483b893e47844a48fedf49aa917d8cf".ToUpper.ToCharArray(), New Converter(Of Char, Byte)(AddressOf CharToBuffer))

        Dim cookieLanguage As System.Net.Cookie = New System.Net.Cookie
        cookieLanguage.Domain = "www.dlapiper.com"
        cookieLanguage.Name = "Language"
        cookieLanguage.Value = UTF8Encoding.ASCII.GetString(buffer)
        cookieLanguage.Path = "/"

        Dim cookieLocalization As System.Net.Cookie = New System.Net.Cookie
        cookieLocalization.Domain = "www.dlapiper.com"
        cookieLocalization.Name = "Localization"
        cookieLocalization.Value = "TimeZone=0&UsesDaylightSavings=False&TimeZoneAbbrev=IDLW&Persists=True"
        cookieLocalization.Path = "/"

        Dim cookieCulture As System.Net.Cookie = New System.Net.Cookie
        cookieCulture.Domain = "www.dlapiper.com"
        cookieCulture.Name = "DefaultCulture"
        cookieCulture.Value = "en-US"
        cookieCulture.Path = "/"

        Dim cookieHideNotice As System.Net.Cookie = New System.Net.Cookie
        cookieHideNotice.Domain = "www.dlapiper.com"
        cookieHideNotice.Name = "hide-cookie-notice"
        cookieHideNotice.Value = "1"
        cookieHideNotice.Path = "/"

        Dim cookieMode As System.Net.Cookie = New System.Net.Cookie
        cookieMode.Domain = "www.dlapiper.com"
        cookieMode.Name = "Mode"
        cookieMode.Value = "1"
        cookieMode.Path = "/"

        Dim cookieNavId As System.Net.Cookie = New System.Net.Cookie
        cookieNavId.Domain = "www.dlapiper.com"
        cookieNavId.Name = "NavId"
        cookieNavId.Value = "1074"
        cookieNavId.Path = "/"

        Dim cookiePortletId As System.Net.Cookie = New System.Net.Cookie
        cookiePortletId.Domain = "www.dlapiper.com"
        cookiePortletId.Name = "PortletId"
        cookiePortletId.Value = "12601"
        cookiePortletId.Path = "/"

        Dim cookieSERVER_PORT As System.Net.Cookie = New System.Net.Cookie
        cookieSERVER_PORT.Domain = "www.dlapiper.com"
        cookieSERVER_PORT.Name = "SERVER_PORT"
        cookieSERVER_PORT.Value = "80"
        cookieSERVER_PORT.Path = "/"

        Dim cookieSiteId As System.Net.Cookie = New System.Net.Cookie
        cookieSiteId.Domain = "www.dlapiper.com"
        cookieSiteId.Name = "SiteId"
        cookieSiteId.Value = "1039"
        cookieSiteId.Path = "/"

        Dim cookieZoneId As System.Net.Cookie = New System.Net.Cookie
        cookieZoneId.Domain = "www.dlapiper.com"
        cookieZoneId.Name = "ZoneId"
        cookieZoneId.Value = "8"
        cookieZoneId.Path = "/"

        Dim cookieEventingStatus As System.Net.Cookie = New System.Net.Cookie
        cookieEventingStatus.Domain = "www.dlapiper.com"
        cookieEventingStatus.Name = "EventingStatus"
        cookieEventingStatus.Value = "1"
        cookieEventingStatus.Path = "/"


        AttorneyList.CookieContainer = New System.Net.CookieContainer()

        AttorneyList.CookieContainer.Add(cookieLanguage)
        AttorneyList.CookieContainer.Add(cookieLocalization)
        AttorneyList.CookieContainer.Add(cookieCulture)
        AttorneyList.CookieContainer.Add(cookieMode)
        AttorneyList.CookieContainer.Add(cookieEventingStatus)
        AttorneyList.CookieContainer.Add(cookieNavId)
        AttorneyList.CookieContainer.Add(cookieSiteId)
        AttorneyList.CookieContainer.Add(cookieSERVER_PORT)
        AttorneyList.CookieContainer.Add(cookieHideNotice)
        AttorneyList.CookieContainer.Add(cookiePortletId)
        AttorneyList.CookieContainer.Add(cookieZoneId)

        'UTF8Encoding.ASCII.GetString(buffer)
        attys = AttorneyList.FindAttorneys("B", "", "", "", "", "", "", "", "", "", UTF8Encoding.ASCII.GetString(buffer), "global", False, "name", 0)
        Response.Write(attys)
    End Sub

    Function CharToBuffer(ByVal character As Char) As Byte
        Return Convert.ToByte(character)
    End Function
End Class