CODEWITHSUNDEEP
perlfork
mswanberg
mswanberg

Reputation: 1285

fork() creates run-time error, am I using it correctly?

I have a Perl program that I've written that parses SQL-like statements and creates a job to run on a mainframe that extracts fields and records based on the criteria. The records are returned to the PC and then output in various formats (e.g. csv, XML, etc.). I co-opted SQL's INTO clause to list the files and outputs.

The issue is that writing records to a SQLite database (one of the output formats) takes a relatively long time and I don't want that to hold up the processing of the next SQL command (in case there are multiple SQL queries passed in). So, I thought I would use fork() to make this happen.

The original bit of code is:

foreach (@into) {
    dointo($_,$recs);
}

@into is a list of files and formats (e.g. 'File1.csv' is a comma-delimited format sent to File1.csv, 'File1.xml' is an XML format written to File1.xml, etc.) to be processed. The subroutine dointo handles each of these. $recs is a sort of iterator that returns records in a variety of formats (flat, anonhash, anonarray, etc.)

So, I changed the code to:

foreach (@into) {
    unless (fork()) {
        dointo($_,$recs);
        exit 0;
    }
}

but now when the code runs, it seems to work, but it pulls a run-time error every time.

I didn't capture the return from fork() because I really don't care about waiting for the forked process to finish. Could this be the error? Does the parent process NEED to wait for the child processes to finish before it can safely exit?

Any help would be appreciated.

(BTW, Windows XP is the OS, Activestate Perl 5.10.0 is the Perl version)

Upvotes: 1

Views: 246

Answers (2)

mob
mob

Reputation: 118705

Don't write to a SQLite database in parallel.

Upvotes: 4

Ovid
Ovid

Reputation: 11677

Fork on Windows has always been a bit dodgy. Remember that fork() can return two false values: 0 if you're in the child process and undef if the fork failed. So for your code above, you may have a failed fork. Also, if you don't want to wait for your children, I think you should set $SIG{CHLD} to 'IGNORE' to avoid creating zombies if you're not waiting.

See perlipc, perlfaq8 and waitpid for more information. Plus, it would help if you said what the runtime error is :)

I forgot to mention that you might also want to look at Parallel::ForkManager. It will make this simpler for you.

And see perlfork to understand limitations of fork emulation on Windows.

Upvotes: 0

Related Questions