CODEWITHSUNDEEP
pythonmultiprocessingpickle
Philipp
Philipp

Reputation: 662

Multiprocessing; How to debug: _pickle.PicklingError: Could not pickle object as excessively deep recursion required

I have a simulation which I can run using Python code and want to create multiple instances of it using a SubProcVecEnv from stable-baselines3. This uses subprocessing to run the simulations on different cores and it was working before I made a number of changes to my code. However, now I receive the error below and do not know how to debug it, because I don't understand which part of my code is causing it. Is there a way to find out which object/ method is causing the recursion depth being exceeded? I also do not remember writing a recursive method anywhere in my code. Researching the error message was not successful.

/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
Traceback (most recent call last):
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 639, in reducer_override
    if sys.version_info[:2] < (3, 7) and _is_parametrized_type_hint(obj):  # noqa  # pragma: no branch
RecursionError: maximum recursion depth exceeded in comparison

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/philipp/Code/ba_pw/train.py", line 84, in <module>
    venv = utils.make_venv(env_class, network, params, remote_ports, monitor_log_dir)
  File "/home/philipp/Code/ba_pw/sumo_rl/utils/utils.py", line 170, in make_venv
    return vec_env.SubprocVecEnv(env_fs)
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 106, in __init__
    process.start()
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/multiprocessing/context.py", line 291, in _Popen
    return Popen(process_obj)
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/multiprocessing/popen_forkserver.py", line 47, in _launch
    reduction.dump(process_obj, buf)
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 372, in __getstate__
    return cloudpickle.dumps(self.var)
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/home/philipp/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 570, in dump
    raise pickle.PicklingError(msg) from e
_pickle.PicklingError: Could not pickle object as excessively deep recursion required.

Upvotes: 0

Views: 4954

Answers (1)

Philipp
Philipp

Reputation: 662

I finally figured out a solution using the answer to this question:

I looks like the object which I want to pickle has too many layers. I called:

sys.setrecursionlimit(3000)

and now it works.

Upvotes: 1

Related Questions