Reputation: 81
My developer recently disappeared and I need to make a small change to my website.
Here's the code I'll be referring to:
my $process = scalar(@rows);
$process = 500 if $process > 500;
my $pm = Parallel::ForkManager->new($process);
This is code from a Perl script that scrapes data from an API system through a cron job. Every time the cron job is run, it's opening a ton of processes for that file. For example, cron-job.pl
will be running 100+ times.
The number of instances it opens is based on the amount of data that needs to be checked so it's different every time, but never seems to exceed 500. Is the code above what would be causing this to happen?
I'm not familiar with using ForkManager, but from the research I've done it looks like it runs the same file multiple times, that way it'll be extracting multiple streams of data from the API system all at the same time.
The problem is that the number of instances being run is significantly slowing down the server. To lower the amount of instances, is it really as simple as changing 500
to a lower number or am I missing something?
Upvotes: 0
Views: 155
Reputation: 386706
Your code simplifies to
my $pm = Parallel::ForkManager->new(500);
It means: Limit the the number of children to 500 at any given time.
If you want fewer children executing any given time, lower that number.
my $pm = Parallel::ForkManager->new(50);
Upvotes: 0
Reputation: 3483
To lower the number of instances created, yes, just lower 500 (in both cases) to something else.
Parallel::ForkManager is a way of using fork (spawning new processes) to handle parallel processing. The parameter passed to new() specifies the maximum number of concurrent processes to create.
Upvotes: 1