CODEWITHSUNDEEP
python-3.xconcurrencyconcurrent.futuresmap-function
Varun Sudarshan
Varun Sudarshan

Reputation: 29

How do i pass objects to executor.map along with iterables?

import glob, os
import tarfile
import concurrent.futures

def function(file1,arc):
    print(file1)
    arc.add(file1)

destination="/home/lol/org"
src=["a","b"]
for i in src:
    if not os.path.exists(os.path.join(destination,i)):
        os.mkdir(os.path.join(destination,i))
#tFile = tarfile.open("files1.tar", 'w')
for i in src:
    name=os.path.join(destination,i,i+".tar")
    tFile = tarfile.open(name, 'w')
    os.chdir(os.path.join("/home",i))
    print(os.getcwd())
    file_list=glob.glob("*.txt")
    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(function,file_list,tFile)
    for f in tFile.getnames():
        print(f)
    tFile.close()

This does not work as the executor.map function is not passing the tFile object to function.How do i fix this?

Upvotes: 2

Views: 1489

Answers (1)

zython
zython

Reputation: 1288

One option you have is to create a list of tFile of length len(file_list) this would require O(n) space and is pretty inefficient.

I'd use itertools.repeat like this, because it is more space saving because it is O(1):

with concurrent.futures.ProcessPoolExecutor() as executor:
    executor.map(function,file_list,itertools.repeat(tFile, len(file_list)))

itertools documentation:

https://docs.python.org/3/library/itertools.html#itertools.repeat

Upvotes: 3

Related Questions