CODEWITHSUNDEEP
pythonshwaitos.system
user3474218
user3474218

Reputation: 41

How to tell python to wait for shell (.sh) script

I am calling a shell (.sh) script from my python code and I want to tell Python to wait for the script to end before continuing to the rest of the code. For the record, the script is calling a HPC cluster some calculations which take approximately 40-50min. I could probably do sleep() and force python to wait for these 40-50min, but firstly I do not always know the amount of time that should wait, and secondly I was hoping for a more efficient way of doing this. So, the script is called by using os.system("bsub < test.sh").

Is there any way to actually tell python wait until the script is finished and then continue with the rest of the code? Thanks in advance

Upvotes: 0

Views: 353

Answers (1)

ti7
ti7

Reputation: 18836

I think @Barmar identifies the problem in a few comments

When you run bsub, it submits the job and immediately returns, rather than waiting for completion.

You should either

  • add the -K arg to bsub for it to wait ref
  • skip bsub and run the script directly
  • write some independent marker at the end of your script (perhaps a file) and have the Python script check for it in a loop (maybe every 1-5s so it doesn't flood that resource)
  • re-write the script in pure Python and directly incorporate it into your logic

Upvotes: 6

Related Questions