MySQL and PDO, speed up query and get result/output from MySQL function (routine)?

Question

Getting the Value:

I've got the levenshtein_ratio function, from here, queued up in my MySQL database. I run it in the following way:

    $stmt = $db->prepare("SELECT r_id, val FROM table WHERE levenshtein_ratio(:input, someval) > 70");
    $stmt->execute(array('input' => $input));
    $result = $stmt->fetchAll(); 

    if(count($result)) {
        foreach($result as $row) {
            $out .= $row['r_id'] . ', ' . $row['val'];
        }
    }

And it works a treat, exactly as expected. But I was wondering, is there a nice way to also get the value that levenshtein_ratio() calculates?

I've tried:

    $stmt = $db->prepare("SELECT levenshtein_ratio(:input, someval), r_id, val FROM table WHERE levenshtein_ratio(:input, someval) > 70");
    $stmt->execute(array('input' => $input));
    $result = $stmt->fetchAll(); 

    if(count($result)) {
        foreach($result as $row) {
            $out .= $row['r_id'] . ', ' . $row['val'] . ', ' . $row[0];
        }
    }

and it does technically work (I get the percentage from the $row[0]), but the query is a bit ugly, and I can't use a proper key to get the value, like I can for the other two items.

Is there a way to somehow get a nice reference for it?

I tried:

$stmt = $db->prepare("SELECT r_id, val SET output=levenshtein_ratio(:input, someval) FROM table WHERE levenshtein_ratio(:input, someval) > 70");

modelling it after something I found online, but it didn't work, and ends up ruining the whole query.

Speeding It Up:

I'm running this query for an array of values:

foreach($parent as $input){
    $stmt = ...
    $stmt->execute...
    $result = $stmt->fetchAll(); 

    ... etc
}

But it ends up being remarkably slow. Like 20s slow, for an array of only 14 inputs and a DB with about 350 rows, which is expected to be in the 10,000's soon. I know that putting queries inside loops is naughty business, but I'm not sure how else to get around it.

EDIT 1

When I use

$stmt = $db->prepare("SELECT r_id, val SET output=levenshtein_ratio(:input, someval) FROM table WHERE levenshtein_ratio(:input, someval) > 70");

surely that's costing twice the time as if I only calculated it once? Similar to having $i < sizeof($arr); in a for loop?

Answer 1

To clean up the column names you can use "as" to rename the column of the function. At the same time you can speed things up by using that column name in your where clause so the function is only executed once.

$stmt = $db->prepare("SELECT r_id, levenshtein_ratio(:input, someval) AS val FROM table HAVING val > 70");

If it is still too slow you might consider a c library like https://github.com/juanmirocks/Levenshtein-MySQL-UDF

doh - forgot to switch "where" to "having", as spencer7593 noted.

Answer 2

I'm assuming that `someval` is an unqalified reference to a column in the table. While you may understand that without looking at the table definition, someone else reading the SQL statement can't tell. As an aid to future readers, consider qualifying your column references with the name of the table or (preferably) a short alias assigned to the table in the statement.

 SELECT t.r_id
      , t.val
   FROM `table` t
  WHERE levenshtein_ratio(:input, t.someval) > 70

That function in the WHERE clause has to be evaluated for every row in the table. There's no way to get MySQL to build an index on that. So there's no way to get MySQL to perform an index range scan operation.

It might be possible to get MySQL to use an index for the query, for example, if the query had an ORDER BY t.val clause, or if there is a "covering index" available.

But that doesn't get around the issue of needing to evaluate the function for every row. (If the query had other predicates that excluded rows, then the function wouldn't necessarily need be evaluated for the excluded rows.)

Adding the expression to the SELECT list really shouldn't be too expensive if the function is declared to be DETERMINISTIC. A second call to a DETERMINISTIC function with the same arguments can reuse the value returned for the previous execution. (Declaring a function DETERMINISTIC essentially means that the function is guaranteed to return the same result when given the same argument values. Repeated calls will return the same value. That is, the return value depends only the argument values, and doesn't depend on anything else.

 SELECT t.r_id
      , t.val
      , levenshtein_ratio(:input, t.someval) AS lev_ratio
   FROM `table` t
  WHERE levenshtein_ratio(:input2, t.someval) > 70

(Note: I used a distinct bind placeholder name for the second reference because PDO doesn't handle "duplicate" bind placeholder names as we'd expect. (It's possible that this has been corrected in more recent versions of PDO. The first "fix" for the issue was an update to the documentation noting that bind placeholder names should appear only once in statement, if you needed two references to the same value, use two different placeholder names and bind the same value to both.)

If you don't want to repeat the expression, you could move the condition from the WHERE clause to the HAVING, and refer to the expression in the SELECT list by the alias assigned to the column.

 SELECT t.r_id
      , t.val
      , levenshtein_ratio(:input, t.someval) AS lev_ratio
   FROM `table` t
 HAVING lev_ratio > 70

The big difference between WHERE and HAVING is that the predicates in the WHERE clause are evaluated when the rows are accessed. The HAVING clause is evaluated much later, after the rows have been accessed. (That's a brief explanation of why the HAVING clause can reference columns in the SELECT list by their alias, but the WHERE clause can't do that.)

If that's a large table, and a large number of rows are being excluded, there might be a significant performance difference using the HAVING clause.. there may be a much larger intermediate set created.

To get an "index used" for the query, a covering index is the only option I see.

 ON `table` (r_id, val, someval)

With that, MySQL can satisfy the query from the index, without needing to lookup pages in the underlying table. All of the column values the query needs are available from the index.

---

FOLLOWUP

To get an index created, we would need to create a column, e.g.

  lev_ratio_foo FLOAT

and pre-populate with the result from the function

UPDATE `table` t
   SET t.lev_ratio_foo = levenshtein_ratio('foo', t.someval) 
;

Then we could create an index, e.g.

... ON `table` (lev_ratio_foo, val, r_id)   

And re-write the query

SELECT t.r_id
     , t.val
     , t.lev_ratio_foo 
  FROM `table` t
 WHERE t.lev_ratio_foo > 70

With that query, MySQL can make use of an index range scan operation on an index with lev_ratio_foo as the leading column.

Likely, we would want to add BEFORE INSERT and BEFORE UPDATE triggers to maintain the value, when a new row is added to the table, or the value of the someval column is modified.

That pattern could be extended, additional columns could be added for values other than 'foo'. e.g. 'bar'

UPDATE `table` t
   SET t.lev_ratio_bar = levenshtein_ratio('bar', t.someval)

Obviously that approach isn't going to be scalable for a broad range of input values.