Process hangs when multiprocessing with XGBoost model batch prediction

Question

Here's a batch prediction case using multiprocessing. Steps:

1. After with mp.Pool(processes=num_processes) as pool, there's a with Dataset(dataset_code) as data in the main process using websocket to get data, and it works well.

2. Then goes to the multiprocessing mission with

result = pool.apply_async(pred_data_from_db, args=(start_index, chunk))

3. Inside pred_data_from_db, users can import their own predict python file flow with

predict = getattr(module, customized_pred_func_name)

4. The issue comes when XGBoost pkl model runs into predict part and it just stops

Other Information:

1. the model has been loaded before multiprocessing starts

2. there is a flask interface for other services to know the progress of the batch prediction

3. this issue will not happen when predicting with an svm pkl model

4. I already set the n_job to 1

I hope I can find ways to locate the problem