CODEWITHSUNDEEP
pythondjangocelerydjango-channels
NMTL
NMTL

Reputation: 101

Celery task can not call Channels consumer via channel_layer.send

I've been setting up a websocket for a fairly simple application. I wrote a JsonWebsocketConsumer as well as some celery tasks in order to perform some longer-running tasks (db accesses). Everything is working without error but my consumers are never getting called. Any ideas?

consumers.py

class NetworkCompConsumer(JsonWebsocketConsumer):
    def connect(self):
        self.accept()

    def disconnect(self, close_code):
        pass

    def receive_json(self, content):
        include = content['include']

        # Reduce Dimensions of Inputs to 2, Just in Case User uploads 3D
        # Geojson
        wkb_w = WKBWriter()
        wkb_w.outdim = 2

        include = GEOSGeometry(json.dumps(include))
        include = GEOSGeometry(wkb_w.write_hex(include))

        print("firing tasks")
        genPolySize.delay(include.json, None, self.channel_name)
        genBuildingCount.delay(include.json, None, self.channel_name)
        # This gets sent
        self.send_json({
            "hello world": "Hi!!!"
        })

    def polygon_area(self, event):
        print("poly area consumer called!!")
        # This does not get sent
        self.send_json(
            {"area": event['value']}
        )

    def building_count(self, event):
        print("building count consumer called!!")
        # This does not get sent
        self.send_json(
            {"buildingCount": event['value']}
        )

tasks.py

def sync_send(channelName, consumer, value):
    channel_layer = get_channel_layer()
    print(channel_layer)
    async_to_sync(channel_layer.send)(
        channelName,
        {
            "type": consumer,
            "value": value,
        }
    )


@shared_task
def genBuildingCount(include, exclude, channelName):
    query_skeleton = building_count_skeleton
    query_skeleton = getQueryTemplate(query_skeleton, exclude is not None, False)
    with connections['gis_data'].cursor() as cursor:
        cursor.execute(
            query_skeleton, [include, exclude] if exclude is not None else [include])
        results = cursor.fetchone()
        buildingCount = results[0]
        sync_send(channelName, "building.count", buildingCount)
        print("successfully completed genBuildingCount")


@shared_task
def genPolySize(include, exclude, channelName):
    query_skeleton = poly_size_skeleton
    if exclude is not None:
        query_skeleton = poly_size_skeleton_exclude
    with connections['gis_data'].cursor() as cursor:
        query_arguments = [
            include,
            exclude
        ] if exclude is not None else [include]
        cursor.execute(query_skeleton, query_arguments)
        results = cursor.fetchone()
        polygonArea = squaredMetersToMiles(results[0])
        sync_send(channelName, "polygon.area", polygonArea)
        print("successfully completed genPolySize")

Logs

django-app_1  | WebSocket HANDSHAKING /ws/network-comparison/ [172.18.0.1:60460]
django-app_1  | WebSocket CONNECT /ws/network-comparison/ [172.18.0.1:60460]
celery_1      | [2020-11-10 18:00:31,916: INFO/MainProcess] Received task: NetworkComparison.tasks.genPolySize[7867638b-76db-4385-b6f6-451970b2f7f3]  
celery_1      | [2020-11-10 18:00:31,921: INFO/MainProcess] Received task: NetworkComparison.tasks.genBuildingCount[875826c0-4c73-4597-9a61-5b6240a4bcfa]  
celery_1      | [2020-11-10 18:00:31,949: WARNING/ForkPoolWorker-3] <channels.layers.InMemoryChannelLayer object at 0x7f0a002c59a0>
celery_1      | [2020-11-10 18:00:31,952: WARNING/ForkPoolWorker-3] successfully completed genPolySize
celery_1      | [2020-11-10 18:00:31,955: INFO/ForkPoolWorker-3] Task NetworkComparison.tasks.genPolySize[7867638b-76db-4385-b6f6-451970b2f7f3] succeeded in 0.03706242493353784s: None
celery_1      | [2020-11-10 18:00:32,050: WARNING/ForkPoolWorker-5] <channels.layers.InMemoryChannelLayer object at 0x7f0a002c9400>
celery_1      | [2020-11-10 18:00:32,053: WARNING/ForkPoolWorker-5] successfully completed genBuildingCount
celery_1      | [2020-11-10 18:00:32,056: INFO/ForkPoolWorker-5] Task NetworkComparison.tasks.genBuildingCount[875826c0-4c73-4597-9a61-5b6240a4bcfa] succeeded in 0.13415803597308695s: None

The celery tasks complete but it appears that polygon_area and building_count are never called. Should I be concerned that the 2 printed channel layers are at a different memory location?

Thanks!

Upvotes: 2

Views: 1393

Answers (1)

NMTL
NMTL

Reputation: 101

My issue was that I was using the In-Memory Channel Layer instead of Redis: https://channels.readthedocs.io/en/stable/topics/channel_layers.html?highlight=get_channel_layer#in-memory-channel-layer

Switching to a Redis channel layer fixed my issue: https://channels.readthedocs.io/en/stable/topics/channel_layers.html?highlight=get_channel_layer#redis-channel-layer

The Celery Worker and Django Worker were in separate docker containers so I guess an in-memory channel layer would never work for this case.

Upvotes: 2

Related Questions