Post Data to ASP .NET page using Delphi and TIdHttp

Question

I have an Asp .net page like this simple one http://issamsoft.com/app2/page1.aspx and I want to post to it some data and extract data from the response, by using TIdHttp. I tried to do that in Delphi2009 like this:

Procedure TForm1.Button1Click(Sender: TObject);
Const
VIEWSTATE = '/wEPDwUKMjA3NjE4MDczNmRkSxPt/LdmgqMd+hN+hkbiqIZuGUk=';
EVENTVALIDATION = '/wEWAwL40NXEDALs0bLrBgKM54rGBtmtdOYy+U7IFq8B25bYT1d4o1iK';
FORMPARAMS = 'TextBox1=Issam&Button1=Button';
URL = 'http://issamsoft.com/app2/page1.aspx';
var
http: TIdHttp;
lstParams: TStringList;
begin
 http := TIdHTTP.Create(self);
 lstParams := TStringList.Create;
 try
  lstParams.Add('__VIEWSTATE='+VIEWSTATE);
  lstParams.Add('__EVENTVALIDATION='+EVENTVALIDATION);
  lstParams.Add(FORMPARAMS);
  http.Request.ContentType := 'application/x-www-form-urlencoded';
  Memo1.Lines.Text := http.Post(url,lstParams);
 finally
  http.Free;
  lstParams.Free;
 end;

end;

but TIdhttp always gives an error(HTTP/1.1 500 Internal Server Error.) I read some comments in the idHttp unit talks about problems with http protocol v 1.1 like this one:

Currently when issuing a POST, IdHTTP will automatically set the protocol to version 1.0 independently of the value it had initially, This is because there are some servers that don't respect the RFC to the full extent. In particular, they don't respect sending/not sending the Expect: 100-Continue header. Until we find an optimum solution that does NOT break the RFC, we will restrict POSTS to version 1.0.

is there something wrong with my code or it's TidHttp Bug? and if the problem is in TIdHttp, is there any workaround? or is there other solution using Indy components?

besides. I've made a solution in C# using WebClient and it works very good.

        private void button1_Click(object sender, EventArgs e)
    {
        WebClient myClient = new WebClient();
        string viewstate = HttpUtility.UrlEncodeUnicode(@"/wEPDwUKMjA3NjE4MDczNmRkSxPt/LdmgqMd+hN+hkbiqIZuGUk=");
        string eventvaildation = HttpUtility.UrlEncodeUnicode(@"/wEWAwL40NXEDALs0bLrBgKM54rGBtmtdOYy+U7IFq8B25bYT1d4o1iK");
        string postdata = "__VIEWSTATE=" + viewstate + "&" + 
            "__EVENTVALIDATION=" + eventvaildation + "&TextBox1=Issam&Button1=Button";
        myClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
        byte[] responce = myClient.UploadData("http://issamsoft.com/app2/page1.aspx", Encoding.ASCII.GetBytes(postdata));
        txtResponse.Text = Encoding.ASCII.GetString(responce);
    } 

where i can find (good/trusted) class like WebClient in Delphi? free preferred :)

Edit: I hope mechanism of VIEWSTATE,EVENTVALIDATION is clear enough for you, they are hash values generated by server, and they may change(already changed), my orginal project has a piece of code just to extract the current VIEWSTATE,EVENTVALIDATION values, but I omit that part just to make my example simple and clear, so when you want to try the above code you must take VIEWSTATE,EVENTVALIDATION values from the current page source.

Answer 1

This is most likely caused by the fact that in C#, you use an ampersand (&) characters to concatenate the VIEWSTATE, EVENTVALIDATION and FORMPARAMS.

But in Delphi, you use a TStringList and Add. Add will not put an ampersand (&), but a CR+LF between the additions.

Hence you are sending different data in Delphi than in C#.

If you change your string concatenation logic in Delphi to match the logic in C#, it should work, not matter what kind of WebClient you use on the Delphi side.

--jeroen

Edit: 20090830

This code works; inspecting the C# and IE7 output using Fiddler2, I found out you also need to escape the '/' and '=' characters.

Edit2: 20090831

Somehow the VIEWSTATE and EVENTVALIDATION on the site have changed since yesterday (though I'm not sure why). I have changed the code, and it works at the time of posting my changes to this answer. This is the new request content as captured per Fiddler when posting the page from within Internet Explorer 7:

__VIEWSTATE=%2FwEPDwUKMjA3NjE4MDczNmRk5%2FC2iWwvlAB3L1wYzRpm3KZhRC0%3D&__EVENTVALIDATION=%2FwEWAwLXzuATAuzRsusGAoznisYGSYOqDGy4vMunY6A8xi6ahQEPI5Q%3D&TextBox1=Issam&Button1=Button

This is the new request as when posted form the Delphi example:

__VIEWSTATE=%2FwEPDwUKMjA3NjE4MDczNmRk5%2FC2iWwvlAB3L1wYzRpm3KZhRC0%3D&__EVENTVALIDATION=%2FwEWAwLXzuATAuzRsusGAoznisYGSYOqDGy4vMunY6A8xi6ahQEPI5Q%3D&TextBox1=Issam&Button1=Button

The below sample now gives this result (note the "Issam" in the result):

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head><title>

</title></head>
<body>
    <form name="form1" method="post" action="page1.aspx" id="form1">
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKMjA3NjE4MDczNg9kFgICAw9kFgICBQ8PFgIeBFRleHQFDVdlbGNvbWUgSXNzYW1kZGSCDMOkTMjkZJgqLkhpK99twpD5+A==" />

<input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWAwKr+O/BBQLs0bLrBgKM54rGBlueO5BU/6BAJMZfHNwh5fsQFuAm" />
    <div>

        <input name="TextBox1" type="text" value="Issam" id="TextBox1" />
        <input type="submit" name="Button1" value="Button" id="Button1" />

        <br />

        <span id="Label1">Welcome Issam</span>
        <br />

    </div>
    </form>
</body>
</html>

Edit3: 20090831

And indeed the VIEWSTATE and EVENTVALIDATION changed again: somehow they keep changing: there seems to be a time factor involved.

So I adapted the code, now it it can extract the VIEWSTATE and EVENTVALIDATION from a GET request, then put that into a POST request.

In addition, it appeared that the '+' and ':' also needs to be escaped. So I dug into the ASP.NET page generation logic, found out that they use HttpUtility.UrlEncode to do the escaping, which ultimately checks HttpUtility.IsSafe which characters to escape. So I adapted took the Reflector reverse engineerd Delphi code into a THttpUtility class.

This is the final code:

{$DEFINE FIDDLER}

unit HttpPostExampleFormUnit;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls, IdBaseComponent, IdComponent, IdTCPConnection, IdTCPClient, IdHTTP;

type
  THttpPostExampleForm = class(TForm)
    Button1: TButton;
    Memo1: TMemo;
    procedure Button1Click(Sender: TObject);
  private
    procedure Log(const What: string; const Content: string);
    procedure LogClear;
  end;

var
  HttpPostExampleForm: THttpPostExampleForm;

implementation

uses
  HttpUtilityUnit;

{$R *.dfm}

procedure AddFormParameter(const StringStream: TStringStream; const ParameterName: string; const ParameterValue: string);
var
  EncodedParameterValue: string;
begin
  StringStream.WriteString(ParameterName);
  StringStream.WriteString('=');
  EncodedParameterValue := THttpUtility.UrlEncode(ParameterValue);
  StringStream.WriteString(EncodedParameterValue);
end;

procedure AddFormParameterSeparator(const StringStream: TStringStream);
begin
  StringStream.WriteString('&');
end;

function ExtractHiddenParameter(const ParameterName: string; const Request: string): string;
//<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKMjA3NjE4MDczNg9kFgICAw9kFgICBQ8PFgIeBFRleHQFDVdlbGNvbWUgSXNzYW1kZGSCDMOkTMjkZJgqLkhpK99twpD5+A==" />
const
  PrefixMask = 'input type="hidden" name="%s" id="%s" value="';
  Suffix = '" />';
var
  Prefix: string;
  PrefixLength: Integer;
  PrefixPosition: Integer;
  SuffixPosition: Integer;
begin
  Prefix := Format(PrefixMask, [ParameterName, ParameterName]);
  PrefixPosition := Pos(Prefix, Request);
  if PrefixPosition = 0 then
    Result := ''
  else
  begin
    PrefixLength := Length(Prefix);
    Result := Copy(Request,
      PrefixPosition + PrefixLength,
      1 + Length(Request) - PrefixPosition - PrefixLength);
    SuffixPosition := Pos(Suffix, Result);
    if SuffixPosition = 0 then
      Result := ''
    else
      Delete(Result, SuffixPosition, 1 + Length(Result) - SuffixPosition);
  end;
end;

procedure THttpPostExampleForm.Button1Click(Sender: TObject);
const
  DefaultVIEWSTATE = '/wEPDwUKMjA3NjE4MDczNmRk5%2FC2iWwvlAB3L1wYzRpm3KZhRC0=';
  DefaultEVENTVALIDATION = '/wEWAwLXzuATAuzRsusGAoznisYGSYOqDGy4vMunY6A8xi6ahQEPI5Q=';
  FORMPARAMS = 'TextBox1=Issam&Button1=Button';
  URL = 'http://issamsoft.com/app2/page1.aspx';
  __VIEWSATE = '__VIEWSTATE';
  __EVENTVALIDATION = '__EVENTVALIDATION';
var
  VIEWSTATE: string;
  EVENTVALIDATION: string;
  getHttp: TIdHttp;
  getRequest: string;
  postHttp: TIdHttp;
  ParametersStringStream: TStringStream;
  postRequest: string;
begin
  LogClear();
  getHttp := TIdHTTP.Create(self);
  try
    getRequest := getHttp.Get(URL);
    Log('GET Request', getRequest);
    VIEWSTATE := ExtractHiddenParameter(__VIEWSATE, getRequest);
    EVENTVALIDATION := ExtractHiddenParameter(__EVENTVALIDATION, getRequest);
    Log('Extracted VIEWSTATE', VIEWSTATE);
    Log('Extracted EVENTVALIDATION', EVENTVALIDATION);
  finally
    getHttp.Free();
  end;

  postHttp := TIdHTTP.Create(self);
{$IFDEF FIDDLER}
  postHttp.ProxyParams.ProxyServer := '127.0.0.1';
  postHttp.ProxyParams.ProxyPort := 8888;
{$ENDIF FIDDLER}
  postHttp.HTTPOptions := postHttp.HTTPOptions + [hoKeepOrigProtocol];
  ParametersStringStream := TStringStream.Create('');
  try
    AddFormParameter(ParametersStringStream, __VIEWSATE, VIEWSTATE);
    AddFormParameterSeparator(ParametersStringStream);
    AddFormParameter(ParametersStringStream, __EVENTVALIDATION, EVENTVALIDATION);
    AddFormParameterSeparator(ParametersStringStream);
    ParametersStringStream.WriteString(FORMPARAMS);
    postHttp.Request.ContentType := 'application/x-www-form-urlencoded';
    ParametersStringStream.Seek(0, soFromBeginning);
    Log('POST Parameters', ParametersStringStream.DataString);
    postRequest := postHttp.Post(url, ParametersStringStream);
    Log('POST Request', postRequest);
  finally
    postHttp.Free;
    ParametersStringStream.Free;
  end;
end;

procedure THttpPostExampleForm.Log(const What, Content: string);
begin
  Memo1.Lines.Add(What + '=');
  Memo1.Lines.Add(Content);
end;

procedure THttpPostExampleForm.LogClear;
begin
  Memo1.Lines.Clear;
end;

end.

And the THttpUtlity class:

unit HttpUtilityUnit;

interface

type
  THttpUtility = class
  private
    class function IsSafe(const ch: Char): boolean;
  public
    class function UrlEncode(s: string): string;
  end;

implementation

uses
  Classes, SysUtils;

class function THttpUtility.IsSafe(const ch: Char): boolean;
begin
  if ((((ch >= 'a') and (ch <= 'z')) or ((ch >= 'A') and (ch <= 'Z'))) or ((ch >= '0') and (ch <= '9'))) then
    Result := True
  else
    case ch of
      '''',
        '(',
        ')',
        '*',
        '-',
        '.',
        '_',
        '!':
        Result := True;
    else
      Result := False;
    end;
end;

class function THttpUtility.UrlEncode(s: string): string;
var
  ch: Char;
  HexCh: string;
  StringStream: TStringStream;
  Index: Integer;
  Ordinal: Integer;
begin
  StringStream := TStringStream.Create('');
  try
    //Note: this is not yet UTF-16 compatible; check before porting to Delphi 2009
    for Index := 1 to Length(s) do
    begin
      ch := s[Index];
      if IsSafe(ch) then
        StringStream.WriteString(Ch)
      else
      begin
        Ordinal := Ord(Ch);
        HexCh := IntToHex(Ordinal, 2);
        StringStream.WriteString('%'+HexCh);
      end;
    end;

    Result := StringStream.DataString;
  finally
    StringStream.Free;
  end;
end;

end.

This should get you going.

--jeroen

Answer 2

You need to update the VIEWSTATE and EVENTVALIDATION values.

Running that code causes a 500 error, but if I open the page and use the new __VIEWSTATE and __EVENTVALIDATION values from the webpage source, it no longer causes a 500 error.

I don't know why a C# client would have worked when the Delphi one didn't, but maybe it doesn't work anymore?

This looks like an ASP issue, not Delphi. Both Indy (TIdHTTP) and Synapse are good socket libraries for Delphi.

Answer 3

If you include the hoKeepOrigProtocol option in the TIdHTTP instance HTTPOptions property, it will not fall back to HTTP 1.0.

This is documented some lines below the comment text which you cited :)

    // If hoKeepOrigProtocol is SET, is possible to assume that the developer
    // is sure in operations of the server
    if not (hoKeepOrigProtocol in FOptions) then begin
      FProtocolVersion := pv1_0;
    end;