Multiprocessing pool not taking list of strings to create individual process

Question

I am trying to write a multiprocessing application where i have list of companies for which individual processes needs to be triggered from a process pool.

I have a function which takes 3 args out of which 1 being self, and the second being a list and third being a company code. I am trying to process the function as a process for each company code. I initially had problem with the self variable, which gives 'pickle' error, which for now i am overcoming by passing None. I have used 'Partial' to multiple arguments problem in multiprocessing, after which i am getting an error "TypeError: can only concatenate str (not "list") to str" when adding a company_list as iterable to my map.

def processing_saved_search_per_company(self, saved_search_list, each_cpy):
    print("Company Key : " + each_cpy)
    print("Saved Search List : " + saved_search_list)

def process(self):
        saved_search_list =[]
        company_list = APICall.fetch_onboarded_companies_from_customer_csv(self)
        saved_search_list_file = os.path.join(code_dir_path, "resources\\saved_search_template.txt")
        try:
            with open(saved_search_list_file, "r") as ss_file_pointer:
                saved_search_list = ss_file_pointer.readlines()
        except IOError as ie:
            print(f"Error Occurred while accessing the Saved Search file reason being :-: {ie}")

        final_ss_list = []
        p=Pool(processes=4)

        #for each_cpy in company_list:
        print("Company List : "+str(company_list))

        func = partial(APICall.processing_saved_search_per_company,None,saved_search_list)
        p.map(func, company_list)
        p.close()

I need the creation of a pool of process which runs like,

p1= processing_saved_search_per_company(self,saved_search_list,"company 1")

p2 = processing_saved_search_per_company(self,saved_search_list,"company 2")

p3 = processing_saved_search_per_company(self,saved_search_list,"company 3")

but getting an error as,

TypeError: can only concatenate str (not "list") to str

Requesting help on this issue.

Thanks, Shahid

Answer 1

Found an workaround to this problem, like passing the Company list as well as part of the partial function and then the array of numbers to the length of Company list to my below function, which would now take index as a new parameter at the end.

processing_saved_search_per_company(self, saved_search_list, companies_list,index):

With the index now coming from the below map, I am able to run the above function for the specific company, without any hassle.

func = partial(APICall.processing_saved_search_per_company,None,saved_search_list, company_list) Index_values =[X for X in range(0,len(companies_list))] p.map(func, index_values) p.close() p.join()

Thanks