CODEWITHSUNDEEP
httpdelphidelphi-xe7
cyberdude
cyberdude

Reputation: 71

what is the best approach to make an http request

what is the best method to do this: i have a txt file filled with web addresses, i have to check all of them using idHTTP component,only a simple check from a web server, downloading the html and finding a match, i want it to to be fast, there is different types of threads and i am not sure what is the best to use, a TParallel for or Task threads or regular threads? i tried before TParallel for and i got stuck at AV, also i've tried Task threads but its not fast, the http request becomes slower by time, i tried also the regular threads and i didnt know how to use it because its complicated to use.
note: Please do NOT downvote i just need advice from the experts. Thank you

Upvotes: 0

Views: 2411

Answers (2)

smooty86
smooty86

Reputation: 1112

You can use TTask and Indy (TIdHTTP). Example:

function GetUrl(const aUrl: string): ITask;
begin
  Result := TTask.Run(
    procedure
    var
      FOutput: string;
      FHTTP: TIdHTTP;
    begin
      FHTTP:=TIdHTTP.Create(nil);
      try
        try
          FOutput:=FHTTP.Get(aUrl);
        except
          // handle errors
        end;
      finally
        FHTTP.Free;
      end;

      TThread.Synchronize(nil,
        procedure
        begin
          ProcessOutput(FOutput); // send your output/result to main thread
        end);
    end );
end;

procedure TForm1.Button1Click(Sender: TObject);
var
  i: Integer;
  list: TStringList;
begin
  list:=TStringList.Create;
  try
    list.LoadFromFile('yourfile.txt');

    // get all your urls
    // you should control how many threads run at the same time
    for i := 0 to list.Count-1 do
      GetUrl(list[i]);
  finally
    list.Free;
  end;
end;

Upvotes: 1

Zam
Zam

Reputation: 2940

First advice: do not use Indy. Use THTTPClient (unit System.Net.HttpClient) -- native for Delphi XE?+

I am still using old TThreads. I could make suggestion only with TThread.

Workflow:

Main thread -- reading your TXT file line by line. After line was readed, you create NEW thread, which are downloading information from WWW.

Sample of application:

unit ufmMain;

interface

uses
  Winapi.Windows, Winapi.Messages,
  System.SysUtils, System.Variants,
  { TThread }
  System.Classes,
  Vcl.Graphics, Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls;

type
  TForm1 = class(TForm)
    Button1: TButton;
    procedure Button1Click(Sender: TObject);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

  TLoad = class(TThread)
  protected
    FURL,
    FOutputFileName: String;

    procedure Execute; override;
  public
    constructor Create(const AURL, AOutputFileName: String); overload;
  end;

  HTTP = class
  public
    class procedure Get(const AURL: String; out AOutputStream: TMemoryStream);
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

uses
  { THTTPClient }
  System.Net.HttpClient;

procedure TForm1.Button1Click(Sender: TObject);
var
  LLoad: TLoad;
  LFile: TextFile;
  LCycle: Integer;
  LUrl: String;
begin
  LCycle := 0;

  AssignFile(LFile, 'urls.txt');
  try
    Reset(LFile);

    while not Eof(LFile) do
      begin
        { Using for generate file name. All file names must be unique }
        Inc(LCycle);

        { Read next URL }
        ReadLn(LFile, LUrl);

        { Create new thread }
        LLoad := TLoad.Create(LUrl, 'Output file No ' + LCycle.ToString + '.htm');
        LLoad.FreeOnTerminate := True;
        LLoad.Start;
      end;
  finally
    CloseFile(LFile);
  end;
end;

{ TLoad }

constructor TLoad.Create(const AURL, AOutputFileName: String);
begin
  inherited Create(True);

  FURL := AURL;
  FOutputFileName := AOutputFileName;
end;

procedure TLoad.Execute;
var
  LResponse: TMemoryStream;
begin
  inherited;

  LResponse := TStringStream.Create;
  try
    HTTP.Get(FURL, LResponse);

    { Save result to file }
    LResponse.SaveToFile(GetCurrentDir + PathDelim + FOutputFileName);
  finally
    LResponse.Free;
  end;
end;

{ HTTP }

class procedure HTTP.Get(const AURL: String; out AOutputStream: TMemoryStream);
var
    LStream: TStream;
  LHTTPClient: THTTPClient;
begin
  LHTTPClient := THTTPClient.Create;
  try
    LStream := LHTTPClient.Get(AURL).ContentStream;

    AOutputStream.CopyFrom(LStream, LStream.Size);
  finally
    LHTTPClient.Free;
  end;
end;

end.

Why I against Indy:

1) THTTPClient do not required additional DLL for works with SSL protocol

2) THTTPClient is modern from Delphi XE8

3) My subjective opinion: THTTPClient works much more smoothly (with less issues) then Indy library. I used Indy for last 10 years, but now all my supported project moved to THTTPClient.

Upvotes: 1

Related Questions