CODEWITHSUNDEEP
pythonpython-3.xconcurrencyconcurrent.futures
Umar
Umar

Reputation: 131

using python concurrent.futures submit with processes numbering

I am trying to iterate over a multiprocess query to website using python 3. I used a modified version of the example code from https://docs.python.org/3/library/concurrent.futures.html example 17.4.2.1. ThreadPoolExecutor Example

    with concurrent.futures.ProcessPoolExecutor(max_workers=self.load) as executor:
        futureThreadVer = {executor.submit(self.get_code(), 60):x for x in range(self.load)}
        for future in concurrent.futures.as_completed(futureThreadVer):
            threadVer = futureThreadVer[future]
            try:
                data=future.result()
            except Exception as exc:
                print("%r generated an exception : %s" %(threadVer, exc))
            else:
                print("%r page data %r" % (self.page, data))

It is a modification of the example code from https://docs.python.org/3/library/concurrent.futures.html example 17.4.2.1. ThreadPoolExecutor Example

Instead of mapping to thread to url, I want to have a process number attached because I am querying the same the url in different processes.

I get the following error:

3 generated an exception : 'int' object is not callable
1 generated an exception : 'int' object is not callable
0 generated an exception : 'int' object is not callable
2 generated an exception : 'int' object is not callable
4 generated an exception : 'int' object is not callable

Upvotes: 1

Views: 1526

Answers (1)

arijeet
arijeet

Reputation: 1874

In your second line:

futureThreadVer = {executor.submit(self.get_code(), 60):x for x in range(self.load)}

The call to executor.submit contains a function call. It should instead be a reference to the function. So, whats happening is instead of calling the function inside the executor, the function gets called in the main thread and self.get_code probably returns an integer which actually gets passed to the executor. The executor then tries to call that integer thinking its a function. This happens because python supports duck typing and the executor expects the first argument to be a function object.

So change your second line to:

futureThreadVer = {executor.submit(self.get_code, 60):x for x in range(self.load)}

Upvotes: 3

Related Questions