CODEWITHSUNDEEP
phpmysqlpdo
Stewart
Stewart

Reputation: 127

PHP PDO Buffered query problem

I'm having some serious problems with the PHP Data Object functions. I'm trying to loop through a sizeable result set (~60k rows, ~1gig) using a buffered query to avoid fetching the whole set.

No matter what I do, the script just hangs on the PDO::query() - it seems the query is running unbuffered (why else would the change in result set size 'fix' the issue?). Here is my code to reproduce the problem:

<?php
$Database = new PDO(
    'mysql:host=localhost;port=3306;dbname=mydatabase',
    'root',
    '',
    array(
        PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
        PDO::MYSQL_ATTR_USE_BUFFERED_QUERY => true
    )
);

$rQuery = $Database->query('SELECT id FROM mytable');

// This is never reached because the result set is too large
echo 'Made it through.';

foreach($rQuery as $aRow) {
    print_r($aRow);
}
?>

If I limit the query with some reasonable number, it works fine:

$rQuery = $Database->query('SELECT id FROM mytable LIMIT 10');

I have tried playing with PDO::MYSQL_ATTR_MAX_BUFFER_SIZE and using the PDO::prepare() and PDO::execute() as well (though there are no parameters in the above query), both to no avail. Any help would be appreciated.

Upvotes: 7

Views: 8825

Answers (3)

Martin
Martin

Reputation: 1

Or maybe you could try mysql functions instead:

while ($row = mysql_fetch_row($query)) {
...
}

Which will definitely be faster, since that foreach statement makes an impression to use fetchAll() instead fetch() each row

Upvotes: -1

Powerlord
Powerlord

Reputation: 88796

If I understand this right, buffered queries involve telling PHP that you want to wait for the entire result set before you begin processing. Prior to PDO, this was the default and you had to call mysql_unbuffered_query if you wanted to deal with results immediately.

Why this isn't explained on the PDO MySQL driver page, I don't know.

Upvotes: 9

user42092
user42092

Reputation:

You could try to split it up into chunks that aren't big enough to cause problems:

<?php    
$id = 0;
$rQuery = $Database->query('SELECT id FROM mytable ORDER BY id ASC LIMIT 100');

do {
    stuff($rQuery);
    $id += 100;
} while ( $rQuery = $Database->query(
            'SELECT id FROM mytable ORDER BY id ASC LIMIT 100 OFFSET '.$id
          )
        );
?>

...you get the idea, anyway.

Upvotes: 2

Related Questions