Receiving "missing FROM-clause" Programming error in django query

Question

I am trying to write a query, that retrieves all subscriptions whose owners should be notified about it's expiry.

I would like to exclude the already notified subscriptions and subscriptions that have a newer subscription available.

The query is next:

Subscription.objects.filter(
    end_date__gte=timezone.now(),
    end_date__lte=timezone.now() + timedelta(days=14),
).exclude(
    Q(notifications__type=Notification.AUTORENEWAL_IN_14) | Q(device__subscriptions__start_date__gt=F('start_date'))
)

Without the | Q(device__subscriptions__start_date__gt=F('start_date') part, the query works perfectly. With it, django (postgres) raises the next error:

django.db.utils.ProgrammingError: missing FROM-clause entry for table "u1"
LINE 1: ...ption" U0 INNER JOIN "orders_subscription" U2 ON (U1."id" = ...

I checked the sql and it seems incorrect:

SELECT "orders_subscription"."id",
       "orders_subscription"."months",
       "orders_subscription"."end_date",
       "orders_subscription"."start_date",
       "orders_subscription"."order_id",
       "orders_subscription"."device_id",
FROM "orders_subscription"
WHERE ("orders_subscription"."churned" = false
       AND "orders_subscription"."end_date" >= '2019-04-05T13:27:39.808393+00:00'::timestamptz
       AND "orders_subscription"."end_date" <= '2019-04-19T13:27:39.808412+00:00'::timestamptz
       AND NOT (("orders_subscription"."id" IN
                   (SELECT U1."subscription_id"
                    FROM "notifications_notification" U1
                    WHERE (U1."type" = 'AUTORENEWAL_IN_2W'
                           AND U1."subscription_id" IS NOT NULL))
                 OR ("orders_subscription"."device_id" IN
                       (SELECT U2."device_id"
                        FROM "orders_subscription" U0
                        INNER JOIN "orders_subscription" U2 ON (U1."id" = U2."device_id")
                        WHERE (U2."start_date" > (U0."start_date")
                               AND U2."device_id" IS NOT NULL))
                     AND "orders_subscription"."device_id" IS NOT NULL)))) LIMIT 21

Execution time: 0.030680s [Database: default]

This is the part that is causing the issue:

INNER JOIN "orders_subscription" U2 ON (U1."id" = U2."device_id")
 WHERE (U2."start_date" > (U0."start_date")
                               AND U2."device_id" IS NOT NULL))

U1 is not defined anywhere (it is locally in the other clause, but that doesn't matter.

The relational model is quite simple, a device can have many subscriptions, a subscription can have many (different) notifications.

class Subscription(models.Model):
    end_date = models.DateTimeField(null=True, blank=True)
    start_date = models.DateTimeField(null=True, blank=True)

    device = models.ForeignKey(Device, on_delete=models.SET_NULL, null=True, blank=True, related_name="subscriptions")
    # Other non significatn fields

class Device(models.Model):
    # No relational fields

class Notification(models.Model):
    subscription = models.ForeignKey('orders.Subscription', related_name="notifications", null=True, blank=True, on_delete=models.SET_NULL)
    # Other non significatn fields

So my question is: is my query wrong or is it a bug in Django ORM query generator?

Answer 1

The ORM clearly fails in translating your clause into SQL. It happens even when the clause is used in isolation (without the preceding clause, i.e. without a U1 alias anywhere in the query).

Apart from the non-existing alias, the ORM also seems to misidentify the origin of F('start_date') – it is the main orders_subscription (the one without an alias), not any aliased table from a subselect.

You can lend the ORM a hand by defining a proper subquery yourself.

(The attempts below are based on the assumption that the intent of the clause is to exclude subscriptions that have sibling subscriptions (= same parent device) with later dates.)

So here's the exclude filter with the corrected clause:

from django.db.models import OuterRef, Subquery

qs.exclude(
    Q(notifications__type=Notification.AUTORENEWAL_IN_14) |
    Q(device_id__in=Subquery(Subscription.objects
        .filter(
            device_id=OuterRef('device_id'), 
            start_date__gt=OuterRef('start_date'))
        .values('device_id')
    ))
)

However, looking at the filter more closely, we are selecting a column (device_id) whose value we have just passed as a filter condition. This is better expressed by an Exists subquery:

from django.db.models import OuterRef, Exists

(qs.annotate(has_younger_siblings=Exists(Subscription.objects.filter(
            device_id=OuterRef('device_id'), 
            start_date__gt=OuterRef('start_date'))))
  .exclude(has_younger_siblings=True)
  .exclude(notifications__type=Notification.AUTORENEWAL_IN_14)
)