CODEWITHSUNDEEP
pythonbashairflow
LinebakeR
LinebakeR

Reputation: 222

How to catch bash script error, which is run through a python script?

I'm using Airflow and i have some python scripts, one of it launch sh script (1 or many). For airflow even if this bash script return an error, it considers that the task is success.

I would like to return the bash error in my python script as airflow will notice.

import glob
import os
from os import path
import subprocess

odsPath = '/apps/data/02_OS/'


for odsFolder in os.listdir(odsPath):
    if(odsFolder in fileNames):
        newPath = odsPath+odsFolder
        print('Script path: ', newPath)
        for foldFiles in os.listdir(newPath):
            if(foldFiles.endswith('sh')):
                print('Sh script can be launch: ', foldFiles)
            if(foldFiles.startswith('run') and foldFiles.endswith('.sh')):
                os.system('sh ' + newPath+'/'+foldFiles)
                print('Script: '+foldFiles+' executed')

In this case, there are bash errors, like this for example 'Number of parameters is incorrect'

Thanks for help :)

Upvotes: 0

Views: 495

Answers (1)

tripleee
tripleee

Reputation: 189749

os.system() runs a command completely outside of Python's control, though you can examine its return code and raise an error if there is a failure. As suggested in the os.system documentation, a better approach which gives you more control and fewer moving parts is subprocess:

subprocess.run(['sh', os.path.join(newPath,foldFiles)],
    check=True)

The check=True causes a subprocess.CalledProcessError to be raised if the subprocess fails. See further Running Bash commands in Python

Upvotes: 3

Related Questions