CODEWITHSUNDEEP
pythonmultithreadingthread-safetypython-importreload
goodGuy
goodGuy

Reputation: 51

Is Python reload thread safe?

Using python, I am writing a nasty cralwer system that cralws something from the websites of each local government, and total websites count to over 100, just in case their webpage changes, I have to use reload to do hot-update. But I am wondering if reload is thread safe. because say, I am reloading moudle Cralwer1 in thread 1, but at the same time, thread 2 is using Cralwer1. Will thread 1's reload cause thread 2 to fail? If so, I have to do a lock or something, otherwise, I can happily do the reload without extra work. Can any one help me out?

Upvotes: 2

Views: 1007

Answers (2)

Raymond Hettinger
Raymond Hettinger

Reputation: 226376

Is Python reload thread safe?

No.

The reload() executes all the pure python code in the module. Any pure python step can thread-switch at any time. So, this definitely isn't safe.

Upvotes: 1

Jay
Jay

Reputation: 2868

reload = re-execute top level code in Crawler1.

Generally speaking without more info/code sample, you can:

  • Encapsulate the "operational" top level code that kicks things off, e.g. put it in a function or a class, and invoke that instead of reloading the whole module. This may involve calling/adding some cleanup function.

  • Use a global variable, which thread1 and thread2 will flip and be aware of to prevent stepping on each other. This doesn't scale quite as well, but can perhaps prevent/delay usage of locks.

  • Using locks is actually not that hard, they even support context managers: https://docs.python.org/3/library/threading.html#with-locks

Upvotes: 0

Related Questions