CODEWITHSUNDEEP
perl
Username
Username

Reputation: 81

Using ForkManager and Perl properly?

My developer recently disappeared and I need to make a small change to my website.

Here's the code I'll be referring to:

my $process = scalar(@rows);
$process = 500 if $process > 500;
my $pm = Parallel::ForkManager->new($process);

This is code from a Perl script that scrapes data from an API system through a cron job. Every time the cron job is run, it's opening a ton of processes for that file. For example, cron-job.pl will be running 100+ times.

The number of instances it opens is based on the amount of data that needs to be checked so it's different every time, but never seems to exceed 500. Is the code above what would be causing this to happen?

I'm not familiar with using ForkManager, but from the research I've done it looks like it runs the same file multiple times, that way it'll be extracting multiple streams of data from the API system all at the same time.

The problem is that the number of instances being run is significantly slowing down the server. To lower the amount of instances, is it really as simple as changing 500 to a lower number or am I missing something?

Upvotes: 0

Views: 155

Answers (2)

ikegami
ikegami

Reputation: 386706

Your code simplifies to

my $pm = Parallel::ForkManager->new(500);

It means: Limit the the number of children to 500 at any given time.

  • If you have fewer than 500 jobs, only that many workers will be created.
  • If you have more than 500 jobs, the manager will start 500 jobs, wait for one to finish, then start the next job.

If you want fewer children executing any given time, lower that number.

my $pm = Parallel::ForkManager->new(50);

Upvotes: 0

Joel
Joel

Reputation: 3483

To lower the number of instances created, yes, just lower 500 (in both cases) to something else.

Parallel::ForkManager is a way of using fork (spawning new processes) to handle parallel processing. The parameter passed to new() specifies the maximum number of concurrent processes to create.

Upvotes: 1

Related Questions