CODEWITHSUNDEEP
pythonsubprocess
NoBullsh1t
NoBullsh1t

Reputation: 686

Why does output of subprocess.run appear too early?

I'm writing a script which runs in a CI/CD pipeline.

It first does some configuration (fetching credentials, pulling down files from a server).

Then, it runs an external CLI tool to analyse these files using subprocess.run(cli_args).

After that is done, the results are published to a few other systems.

My problem is that the output of the CLI appears before some or all of the previous logs. A simplified version of the code may look like this:

print("Fetching CLI configuration")
exit_code, config = fetch_cli_configuration_without_ssl_verification(server_credentials)
if exit_code != 0 || not config:
    print("Error; something went wrong")
    exit(exit_code)
print("Got CLI configuration")

print("Loading result server credentials")
res_server = os.environ["RES_SERVER"]
res_user = os.environ["RES_USER"]
res_pwd = os.environ["RES_PWD"]
if not (res_server && res_user && res_pwd):
    print("Could not load result server credentials")
    exit(1)
print("Loaded credentials for result server", res_server)

print("Running CLI-Tool")
# Logs "I am the CLI"
exit_code = subprocess.run(["mycli", "do", "this", "and", "that"], cwd="somesubdir").returncode
if exit_code != 0:
   print("Error: CLI tool finished with non-zero exit code")
   exit(exit_code)
print("CLI tool finished successfully")

print("Uploading result data")
upload_result_data(read_file_text("cli_results.json"), res_server, res_user, res_pwd)
print("Done uploading result data")

The output I get looks something like

/opt/python-3.10.4/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'result_server.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
I am the CLI
Fetching CLI configuration
Got CLI configuration
Loading result server credentials
Loaded credentials for result server result_server.com
Running CLI-Tool
CLI tool finished successfully
Uploading result data
Done uploading result data

How can I make sure the CLI output appears after "Running CLI-Tool"?

Upvotes: 2

Views: 188

Answers (1)

Ahmed AEK
Ahmed AEK

Reputation: 18229

This is most likely because python buffers its standard output, so flush it before calling subprocess

sys.stdout.flush()

Upvotes: 2

Related Questions