CODEWITHSUNDEEP
pythonpython-3.xpython-multiprocessingpython-asyncio
Desiigner
Desiigner

Reputation: 2316

Asyncio and multiprocessing.Process- how to pass a coroutine?

I've got a coroutine which is scraping some data. I'm trying to add multiprocessing to speed it up. I've come up with using multiprocesesing.Process. My coroutine is called fetch_students_pages

What I'm trying to do is to pass this coroutine as target to Process class:

    def fetch_students_concurrently(self):
        for _ in range(10):
            Process(target=self.fetch_students_pages).start()

but if fails with:

/usr/lib/python3.7/multiprocessing/process.py:99: RuntimeWarning: coroutine 'StudentPageParser.fetch_students_pages' was never awaited self._target(*self._args, **self._kwargs) RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Is there a way to await it or use another solution instead?

Upvotes: 2

Views: 1336

Answers (1)

J_H
J_H

Reputation: 20450

Consider using map() instead:

cores = 8

with Pool(cores) as p:
    p.map(self.fetch_students_pages, range(cores))

Upvotes: 2

Related Questions