CODEWITHSUNDEEP
sqlperformancepostgresql
David Eyk
David Eyk

Reputation: 12541

Why is this DISTINCT/INNER JOIN/ORDER BY postgresql query so slow?

This query takes ~4 seconds to complete:

SELECT DISTINCT "resources_resource"."id",
                  "resources_resource"."heading",
                  "resources_resource"."name",
                  "resources_resource"."old_name",
                  "resources_resource"."clean_name",
                  "resources_resource"."sort_name",
                  "resources_resource"."see_also_id",
                  "resources_resource"."referenced_passages",
                  "resources_resource"."resource_type",
                  "resources_resource"."ord",
                  "resources_resource"."content",
                  "resources_resource"."thumb",
                  "resources_resource"."resource_origin"
  FROM "resources_resource"
  INNER JOIN "resources_passageresource" ON ("resources_resource"."id" = "resources_passageresource"."resource_id")
  WHERE "resources_passageresource"."start_ref" >= 66001001
  ORDER BY "resources_resource"."ord" ASC, "resources_resource"."sort_name" ASC LIMIT 5

By popular request, EXPLAIN ANALYZE:

Limit  (cost=1125.50..1125.68 rows=5 width=803) (actual time=4434.076..4434.557 rows=5 loops=1)
  ->  Unique  (cost=1125.50..1136.91 rows=326 width=803) (actual time=4434.076..4434.557 rows=5 loops=1)
        ->  Sort  (cost=1125.50..1126.32 rows=326 width=803) (actual time=4434.075..4434.075 rows=6 loops=1)
              Sort Key: resources_resource.ord, resources_resource.sort_name, resources_resource.id, resources_resource.heading, resources_resource.name, resources_resource.old_name, resources_resource.clean_name, resources_resource.see_also_id, resources_resource.referenced_passages, resources_resource.resource_type, resources_resource.content, resources_resource.thumb, resources_resource.resource_origin
              Sort Method:  quicksort  Memory: 424kB
              ->  Hash Join  (cost=697.00..1111.89 rows=326 width=803) (actual time=3.453..41.429 rows=424 loops=1)
                    Hash Cond: (resources_passageresource.resource_id = resources_resource.id)
                    ->  Bitmap Heap Scan on resources_passageresource  (cost=10.78..190.19 rows=326 width=4) (actual time=0.107..0.401 rows=424 loops=1)
                          Recheck Cond: (start_ref >= 66001001)
                          ->  Bitmap Index Scan on resources_passageresource_start_ref  (cost=0.00..10.70 rows=326 width=0) (actual time=0.086..0.086 rows=424 loops=1)
                                Index Cond: (start_ref >= 66001001)
                    ->  Hash  (cost=431.32..431.32 rows=2232 width=803) (actual time=3.228..3.228 rows=2232 loops=1)
                          Buckets: 1024  Batches: 2  Memory Usage: 947kB
                          ->  Seq Scan on resources_resource  (cost=0.00..431.32 rows=2232 width=803) (actual time=0.002..1.621 rows=2232 loops=1)
Total runtime: 4435.460 ms

This is ORM-generated SQL. I can work in SQL, but I'm definitely not proficient, and the EXPLAIN output here is mystifying to me. What about this query is dragging me down?

UPDATE: @Ybakos identified that the ORDER_BY was causing trouble. Removing the ORDER_BY clause altogether helps a bit, but the query still takes 800ms. Here's the EXPLAIN ANALYZE, sans ORDER_BY:

HashAggregate  (cost=1122.49..1125.75 rows=326 width=803) (actual time=787.519..787.559 rows=104 loops=1)
  ->  Hash Join  (cost=697.00..1111.89 rows=326 width=803) (actual time=3.381..7.312 rows=424 loops=1)
        Hash Cond: (resources_passageresource.resource_id = resources_resource.id)
        ->  Bitmap Heap Scan on resources_passageresource  (cost=10.78..190.19 rows=326 width=4) (actual time=0.095..0.686 rows=424 loops=1)
              Recheck Cond: (start_ref >= 66001001)
              ->  Bitmap Index Scan on resources_passageresource_start_ref  (cost=0.00..10.70 rows=326 width=0) (actual time=0.079..0.079 rows=424 loops=1)
                    Index Cond: (start_ref >= 66001001)
        ->  Hash  (cost=431.32..431.32 rows=2232 width=803) (actual time=3.173..3.173 rows=2232 loops=1)
              Buckets: 1024  Batches: 2  Memory Usage: 947kB
              ->  Seq Scan on resources_resource  (cost=0.00..431.32 rows=2232 width=803) (actual time=0.002..1.568 rows=2232 loops=1)
Total runtime: 787.678 ms

Upvotes: 1

Views: 3284

Answers (3)

Andriy M
Andriy M

Reputation: 77687

It seems to me, DISTINCT has to be used to remove duplicates produced by the join. So my question is, why produce the duplicates in the first place? I'm not entirely sure what this query's being ORM-generated must imply, but if rewriting it is an option, you could certainly rewrite it in such a way as to prevent duplicates from appearing. For instance, using IN:

SELECT "resources_resource"."id",
       "resources_resource"."heading",
       "resources_resource"."name",
       "resources_resource"."old_name",
       "resources_resource"."clean_name",
       "resources_resource"."sort_name",
       "resources_resource"."see_also_id",
       "resources_resource"."referenced_passages",
       "resources_resource"."resource_type",
       "resources_resource"."ord",
       "resources_resource"."content",
       "resources_resource"."thumb",
       "resources_resource"."resource_origin"
  FROM "resources_resource"
  WHERE "resources_resource"."id" IN (
        SELECT "resources_passageresource"."resource_id"
          FROM "resources_passageresource"
          WHERE "resources_passageresource"."start_ref" >= 66001001
        )
  ORDER BY "resources_resource"."ord" ASC, "resources_resource"."sort_name" ASC LIMIT 5

or using EXISTS:

SELECT "resources_resource"."id",
       "resources_resource"."heading",
       "resources_resource"."name",
       "resources_resource"."old_name",
       "resources_resource"."clean_name",
       "resources_resource"."sort_name",
       "resources_resource"."see_also_id",
       "resources_resource"."referenced_passages",
       "resources_resource"."resource_type",
       "resources_resource"."ord",
       "resources_resource"."content",
       "resources_resource"."thumb",
       "resources_resource"."resource_origin"
  FROM "resources_resource"
  WHERE EXISTS (
        SELECT *
          FROM "resources_passageresource"
          WHERE "resources_passageresource"."resource_id" = "resources_resource"."id"
            AND "resources_passageresource"."start_ref" >= 66001001
        )
  ORDER BY "resources_resource"."ord" ASC, "resources_resource"."sort_name" ASC LIMIT 5

And, of course, if it's acceptable to rewrite the query completely, I would also remove the long table names in front of column names. Consider the following, for instance (the IN query rewritten):

SELECT "id",
       "heading",
       "name",
       "old_name",
       "clean_name",
       "sort_name",
       "see_also_id",
       "referenced_passages",
       "resource_type",
       "ord",
       "content",
       "thumb",
       "resource_origin"
  FROM "resources_resource"
  WHERE "resources_resource"."id" IN (
        SELECT "resource_id"
          FROM "resources_passageresource"
          WHERE "start_ref" >= 66001001
        )
  ORDER BY "ord" ASC, "sort_name" ASC LIMIT 5

Upvotes: 2

Andrew Lazarus
Andrew Lazarus

Reputation: 19330

That seems like a long time in the JOIN. The default memory settings in postgresql.conf are too low for any modern computer. Have you remembered to bump them up?

Upvotes: 1

ybakos
ybakos

Reputation: 8630

It's the combination of ORDER BY with LIMIT.

If you don't have an index on (ord, sort_name) then I bet this is the cause of the slow performance. Or perhaps an index on (start_ref, ord, sort_name) is necessary for this particular query. Lastly, due to that join, perhaps have the left/first table be the one upon which your ORDER BY criteria applies.

Upvotes: 1

Related Questions