CODEWITHSUNDEEP
multithreadingdelphiparallel-processingthreadpooldelphi-10.3-rio
maf-soft
maf-soft

Reputation: 2552

Delphi TThreadPool: wait for free thread slot before proceeding with code

I know TTask and have used TTask.WaitForAll(array) successfully, as well as TParallel.&For().

But now I want to do a seemingly simple thing and don't find out how:

I have an unknown number of items coming in, this can be millions or only a few, and I don't know in advance. How can I work on them in parallel (just about 4 threads or so), but without a queue? If max threads are already busy, I want to wait for the next free slot. Something like a TTask.Run() which just doesn't come back until it really starts running.

I guess I'm just overseeing something simple...?

When I'm through, I want to wait for all remaining tasks to finish. But of course I don't want to have millions of them in an array for WaitForAll().

I can imagine a possible solution (but I don't like it and hope for a much easier one using TTask or similar):

I know this might be the preferred way anyway in some cases, but my situation would not profit from it (like reusing any objects, connections or so).

Pseudocode of what would be nice and clean:

MyThreadPool:= TMyThreadPool.Create(4);
while GetNextItem(out Item) do
  //the following comes back when it has really been started:
  MyThreadPool.Run(procedure begin Work(Item); end);
MyThreadPool.WaitFor;

Upvotes: 2

Views: 1161

Answers (1)

maf-soft
maf-soft

Reputation: 2552

This seems to be a working solution, but abusing TParallel.&For is maybe not really nice. I'm still hoping for a better answer.

if FindFirst(Path, 0, SearchRec) = 0 then
  try
    TParallel.&For(0, 99999,
      procedure(I: Integer; LoopState: TParallel.TLoopState)
      var
        Filename: string;
      begin
        if LoopState.ShouldExit then Exit;

        TMonitor.Enter(Self);
        try
          Filename:= SearchRec.Name;
          if (FindNext(SearchRec) <> 0) or TThread.CheckTerminated then
            LoopState.Stop; //or .Break?
        finally
          TMonitor.Exit(Self);
        end;

        try
          ProcessFile(Filename);
        except
          on E: Exception do Log(E.ToString); //maybe also want to Stop
        end;
      end);
  finally
    FindClose(SearchRec);
  end;

I wrote a lot of trace logs and it looks good. The only bad thing is that after the last file it still starts 10-20 more executions which are then exited at the beginning.

It also seems the default threadpool cannot be restricted to less than the number of processors.

Please comment if you think anything is bad or can/should be improved.

Upvotes: 2

Related Questions