Indy Proxy Server HTTPS to HTTP

Question

I am trying to write a proxy server in Indy to receive HTTPS calls from external clients and forward them in HTTP to another server application on the same machine. Reason is that the other application does not support SSL and so I want to wrap its traffic in an SSL layer for external security.

My current approach is to use a TIdHTTPserver with an SSL IOHandler, and in its OnCommandGet handler I create a TIdHTTP client on the fly which fetches a TFileStream ContentStream from the internal application and returns that stream as the Response.ContentStream to the external caller.

The problem with this approach is the latency caused by having to wait for the internal content stream to be fully received before the external stream can start to be sent. And for example it cannot work for streaming media.

My question is: is there a better way to proxy HTTPS to HTTP that would work for streams? I.e without having to use an intermediate file stream.

Answer 1

Thank you for the very comprehensive answer. The way I finally solved it was to create a TStream descendent to use as the ContentStream for the server response. The TStream wis a wrapper around a TIdTcpClient which contains a rudimentary HTTP implementation and whose TStream.Read function fetches the HTTP contents for the Tcp connection.

type
  TTcpSocketStream = class(TStream)
  private
    FAuthorization: string;
    FBuffer: TBytes;
    FBytesRead: Int64;
    FCommand: string;
    FContentLength: Int64;
    FContentType: string;
    FDocument: string;
    FHeaders: TIdHeaderList;
    FHost: string;
    FIntercept: TServerLogEvent;
    FPort: Integer;
    FResponseCode: Integer;
    FQueryParams: string;
    FTcpClient: TIdTCPClient;
    FWwwAuthenticate: string;
  public
    constructor Create;
    destructor Destroy; override;
    procedure Initialize;
    function Read(var Buffer; Count: Longint): Longint; override;
    function Seek(const Offset: Int64; Origin: TSeekOrigin): Int64; override;
    property Authorization: string read FAuthorization write FAuthorization;
    property Command: string read FCommand write FCommand;
    property ContentType: string read FContentType;
    property ContentLength: Int64 read FContentLength;
    property Document: string read FDocument write FDocument;
    property Host: string read fHost write FHost;
    property Intercept: TServerLogEvent read FIntercept write FIntercept;
    property Port: Integer read FPort write FPort;
    property QueryParams: string read FQueryParams write FQueryParams;
    property ResponseCode: Integer read FResponseCode;
    property WWWAuthenticate: string read FWwwAuthenticate 
      write FWwwAuthenticate;
  end;

const
  crlf = #13#10;
  cContentSeparator = crlf+crlf;

implementation

{ TTcpSocketStream }

constructor TTcpSocketStream.Create;
begin
  inherited;

  FHeaders := TIdHeaderList.Create(QuoteHTTP);
  FTcpClient := TIdTcpClient.Create(nil);
  FTcpClient.ConnectTimeout := 5000;
  FTcpClient.ReadTimeout := 5000;

  FCommand := 'GET';
  FPort := 443;
  FResponseCode := 404;
end;

destructor TTcpSocketStream.Destroy;
begin
  if FTcpClient.Connected then
    FTcpClient.Disconnect;

  if FTcpClient.Intercept <> nil then
  begin
    FTcpClient.Intercept.Free;
    FTcpClient.Intercept := nil;
  end;

  FTcpClient.Free;
  FHeaders.Free;
  SetLength(FBuffer, 0);

  inherited;
end;

procedure TTcpSocketStream.Initialize;
var
  s: string;
  LLog: TClientLogEvent;
  LRespText: string;
begin
  try
    if FQueryParams <> '' then
      FQueryParams := '?' + FQueryParams;

    FTcpClient.Port := FPort;
    FTcpClient.Host := FHost;

    if FIntercept <> nil then
    begin
      LLog := TClientLogEvent.Create;
      LLog.OnLog := FIntercept.OnLog;
      FTcpClient.Intercept := LLog;
    end;

    FTcpClient.Connect;
    if FTcpClient.Connected then
    begin

      FTcpClient.IOHandler.Writeln(Format('%s %s%s HTTP/1.1', 
        [FCommand, FDocument, FQueryParams]));
      FTcpClient.IOHandler.Writeln('Accept: */*');
      if FAuthorization <> '' then
        FTcpClient.IOHandler.Writeln(Format('Authorization: %s',
          [FAuthorization]));
      FTcpClient.IOHandler.Writeln('Connection: Close');
      FTcpClient.IOHandler.Writeln(Format('Host: %s:%d', [FHost, FPort]));
      FTcpClient.IOHandler.Writeln('User-Agent: Whitebear SSL Proxy');
      FTcpClient.IOHandler.Writeln('');

      LRespText := FTcpClient.IOHandler.ReadLn;
      s := LRespText;
      Fetch(s);
      s := Trim(s);
      FResponseCode := StrToIntDef(Fetch(s, ' ', False), -1);

      repeat
        try
          s := FTcpClient.IOHandler.ReadLn;
        except
          on Exception do
            break;
        end;
        if s <> '' then
          FHeaders.Add(s);
      until s = '';

      FContentLength := StrToInt64Def(FHeaders.Values['Content-Length'], -1);
      FContentType := FHeaders.Values['Content-Type'];
      FWwwAuthenticate := FHeaders.Values['WWW-Authenticate'];
    end;

  except
    on E:Exception do ;
  end;
end;

function TTcpSocketStream.Read(var Buffer; Count: Integer): Longint;
begin
  Result := 0;
  try
    if FTcpClient.Connected then
    begin
      if Length(FBuffer) < Count then
        SetLength(FBuffer, Count);
      FTcpClient.IOHandler.ReadBytes(FBuffer, Count, False);
      Move(FBuffer[0], PChar(Buffer), Count);
      Inc(FBytesRead, Count);
      Result := Count;
    end;
  except
    on Exception do ;
  end;
end;

function TTcpSocketStream.Seek(const Offset: Int64; Origin: TSeekOrigin): Int64;
begin
  Result := 0;
  case Origin of
    soBeginning: Result := Offset;
    soCurrent: Result := FBytesRead + Offset;
    soEnd: Result := FContentLength + Offset;
  end;
end;

Answer 2

If the requesting client supports HTTP 1.1 chunking (see RFC 2616 Section 3.6.1), that would allow you to read data from the target server and send it immediately to the client in real-time.

If you are using a fairly recent version of Indy, TIdHTTP has an OnChunkReceived event, and an hoNoReadChunked flag in its HTTPOptions property:

New TIdHTTP flags and OnChunkReceived event

In your TIdHTTPServer.OnCommand... event handler, you can populate the AResponseInfo as needed. Make sure to:

• leave AResponseInfo.ContentText and AResponseInfo.ContentStream unassigned

• set AResponseInfo.ContentLength to 0

• set AResponseInfo.TransferEncoding to 'chunked'

Then call the AResponseInfo.WriteHeader() method directly, such as in the TIdHTTP.OnHeadersRecceived event, to send the response headers to the client.

Then you can read the target server's response body using OnChunkedReceived or hoNoReadChunked, and write each received chunk to the client using AContext.Connection.IOHandler directly.

However, there are some caveats to this:

• if you use the TIdHTTP.OnChunkReceived event, you still need to provide an output TStream to TIdHTTP or else the event is not triggered (this restriction might be removed in a future release). However, you can use TIdEventStream without assigning an OnWrite event handler to it. Or write a custom TStream class that overrides the virtual Write() method to do nothing. Or, just use any TStream you want, and have the OnChunkReceived event handler clear the received Chunk so nothing is available to write to the TStream.

• if you use the hoNoReadChunked flag, this allows you to manually read the HTTP chunks from TIdHTTP.IOHandler directly after TIdHTTP exits. Just make sure you enable HTTP keep-alives or else TIdHTTP will close the connection to the server before you have a chance to read the server's response body.

If you are using an older version of Indy, or if the target server does not support chunking, all is not lost. You should be able to write a custom TStream class that overwrites the virtual Write() method to write the provided data block to the client as an HTTP chunk. And then you can use that class as the output TStream for TIdHTTP.

If the client does not support HTTP chunking, or if these approaches do not work for you, then you will likely have to resort to using TIdTCPServer directly instead of TIdHTTPServer and implement the entire HTTP protocol yourself from scratch, then you can handle your own streaming as needed. Have a look at the source code for TIdHTTPProxyServer for some ideas (TIdHTTPProxyServer itself is not suitable for your particular situation, but it will show you how to pass HTTP requests/responses between connections in near real-time in general).