Break up memory hogging PHP script with nested for loops

Question

I am trying to build a table of part numbers with this script. I am quickly running out of memory. I have commented out the loops I couldn't even get to:

 foreach ($mat_codes as $mat_code) {
     for ($n = 0; $n < count($shape_codes); $n++){
        for ($d1 = 0; $d1 <= 12; $d1++){
          for ($d1_dec = 1; $d1_dec <= 16; $d1_dec++){
            for ($d2 = 0; $d2 <= 12; $d2++){
              //for ($d2_dec = 1; $d2_dec <= 16; $d2_dec++){
                $build_part = (object) NULL;
                $build_part = $mat_code->part_code;
                $build_part .= $shape_codes[$n]->part_code;
                $build_part .= '.';
                if ($d1 != 0) {
                  $build_part .=  $d1 . '-';
                }
                $build_part .= $d1_dec;
                $build_part .= '.';
                //  if ($d2 != 0) {
                //    $build_part .=  $d2 . '-';
                //  }
                // $build_part .= $d2_dec;
                // $build_part .= '.';

                // * save $build_part to MYSQL * //
             //}
           }
        }
      }
    }
  }

Unsurprisingly this is returning

PHP Fatal error: Allowed memory size of 134217728 byets exhauseted (tried to allocate 71 bytes)

For the record there are about 16 values in $mat_codes and 16 in $shape_codes. By my calculations, I'm trying to create 692,224 part numbers in this example alone. I think I'm going to add more and have up to ~34 million. IMHO, it would be silly to try and get this done by just allocating more memory to the job. My question -- of course -- how do I break this script up into smaller chunks?

Edit: I have a workaround. I apologize it is out of the scope I defined in the original question. I didn't mention that this is simply a table of potential part numbers. Particularly, all the potential part numbers allowed by the logic of the part numbering system. Rather than have a master list of part numbers at the ready just waiting to be used I figure I might be better off validating new part input against the part numbering logic, a function like could_this_be_a_part_number($input). It could just test the inputted string against the part numbering logic as opposed to having some master table I would need to check against.

Edit x2: I'm sorry this is the worst. I moved unset($build_part) into the innermost for loop. The problem was I was trying to a development function in Drupal to print out each $build_part before it was unset - dpm($build_part). The script runs fine, its trying to build a webpage with that many part numbers printed to it is what breaks the script.

Answer 1

First of all: $build_part seems like it's not supposed to be an object, because you treat it like a string, so don't initialize it like that.

$build_part = NULL; will suffice.

Second of all: We don't know how you'd like your data to be saved in your database, but you could (as my original answer somehow proposed) build up a query, send it to the database and then continue.

Now let's have a look at your code:

$i = 0;
foreach ($mat_codes as $mat_code) {
     for ($n = 0; $n < count($shape_codes); $n++){
        for ($d1 = 0; $d1 <= 12; $d1++){
          for ($d1_dec = 1; $d1_dec <= 16; $d1_dec++){
            for ($d2 = 0; $d2 <= 12; $d2++){
              for ($d2_dec = 1; $d2_dec <= 16; $d2_dec++){
                $i++;
                $build_part = (object) NULL;
                $build_part = $mat_code->part_code.".";
                $build_part .= $shape_codes[$n]->part_code;
                $build_part .= '.';
                if ($d1 != 0) {
                  $build_part .=  $d1 . '-';
                }
                $build_part .= $d1_dec;
                $build_part .= '.';
                 if ($d2 != 0) {
                   $build_part .=  $d2 . '-';
                 }
                $build_part .= $d2_dec;
                // echo "$i $build_part <br>";
                // * save $build_part to MYSQL * //
             }
             // This is first triggered when the count reaches 16 items
             echo "--- lvl 5<br>";
           }
           // 208 items
           echo "--- lvl 4<br>";
        }
        // 3328 items
        echo "--- lvl 3<br>";
      }
      echo "--- lvl 2<br>";
    }
    echo "--- lvl 1<br>";
}

I have determined (and you can do it somewhere else) the level 3 sign as the point where you could be sending some pre-constructed MySQL query to the database. Something like:

$mysqli->query("INSERT INTO parts (part_id) VALUES $very_long_string;");

Also look at this MySQL command: INSERT DELAYED, I think maybe it might (sic!) be quite useful in your situation.

$very_long_string would be built in the most inner loop like:

$very_long_string .= "('$build_part'),";

and in the end (before we create and send the final query) we would cut the last comma:

$very_long_string = rtrim($very_long_string, ",");

Then send the 3000+ items long query into the database. Then unset($very_long_string) and start again.

So I guess the final code would look something like this:

foreach ($mat_codes as $mat_code) {
for ($n = 0; $n < count($shape_codes); $n++){
    for ($d1 = 0; $d1 <= 12; $d1++){
        $very_long_string = null;
        for ($d1_dec = 1; $d1_dec <= 16; $d1_dec++){
            for ($d2 = 0; $d2 <= 12; $d2++){
                for ($d2_dec = 1; $d2_dec <= 16; $d2_dec++){
                $build_part = NULL;
                $build_part = $mat_code->part_code.".";
                $build_part .= $shape_codes[$n]->part_code;
                $build_part .= '.';
                if ($d1 != 0) {
                    $build_part .=  $d1 . '-';
                }
                $build_part .= $d1_dec;
                $build_part .= '.';
                if ($d2 != 0) {
                    $build_part .=  $d2 . '-';
                }
                $build_part .= $d2_dec;
                $very_long_string .= "('$build_part'),";
                }
            }
        }
        $very_long_string = rtrim($very_long_string, ",");
        $mysqli->query("INSERT INTO parts (part_id) VALUES $very_long_string;");
        unset($very_long_string);
    }
}
}

EDIT after reading your EDIT: I was of course dealing with the fact that you wanted to populate the database :D .. If you need a validating function, then that's a whole another story.