CODEWITHSUNDEEP
pythonprogress-bartqdm
nichollsg
nichollsg

Reputation: 390

How to set up progress bar and also print output from subprocess.Popen

I am trying to use a progress bar to reduce the verbosity of running an FPGA build script when using python subprocess. The build script can take hours to run and the build output is extremely verbose. Currently, I print the output of the script for the user to see that progress is being made and also get an idea of how far along it is. I would like to clean this up for users because the verbosity masks important python logs and it's just really annoying.

In essence, I'm hoping to use tqdm to indicate progress. What I want is this (it doesn't have to match the example below perfectly):

When the build starts, the terminal might look like this:

INFO: starting build
Running Build [                           ] (0:00:01.2345)
@I: compiling file <x>

When it's done, it would look something like this (note that terminal output line is gone)

INFO: starting build
Running Build [...........................] (6:25:48.6251)
<python continues logging stuff here>

To determine the progress, I will check subprocess.Popen.stdout to determine if it's a line in the build script. Here is some pseudo-code to show what I'm trying to do:

from subprocess import Popen, PIPE

pb = <initialize progress bar>

lines_in_build_script = [
  "step 1",
  "step 2",
]

p = Popen(cmd, shell=True, stdout=PIPE)
for line in p.stdout:
  pb.update_build_output(line)
  if line in lines_in_build_script:
    pb.increment_progress() # I would prefer if it was just a spinning wheel because looking for line in output is asking for trouble
rc = p.wait()

EDIT: tqdm isn't required, that's just a package I'm aware of. I'm open to any suggestions about how to tackle this problem.

Upvotes: 1

Views: 3533

Answers (3)

rsalmei
rsalmei

Reputation: 3605

author of alive-progress here!

Actually, alive-progress does support placing texts above it, there's not much sense putting them below, since lines on a terminal run from top to bottom, so the bar has to leave behind all lines printed, like a log.

Today I've released the hiding of the final receipt, and together with spinner hiding, which was already supported, you could do it like this:

simulate command with elapsed

And yes, all with a great live spinner!

I think it is very cool to show the elapsed time, but you can also remove it if you'd like, as well as the "Please wait" message, like this:

simulate command with spinner only

Upvotes: 1

nichollsg
nichollsg

Reputation: 390

I wasn't happy with the tqdm or alive_progress capability. I found yaspin and created a custom "title" formatter since setting the percent of packages was impossible to characterize based on the steps I use to determine the progress. Instead, I add a counter to a yaspin formatter along with some methods to update the text manually. Not exactly what I wanted, but it definitely streamlined stdout:

class BarFormat:
    def __init__(self, title: str, total=None, width=None):
        self.title = title
        self._total = total
        self._cnt = 0
        self._start = datetime.datetime.now()
        self._status = ""
        self._width = width if width else os.get_terminal_size(0)[0]

    def __str__(self):
        delta = datetime.datetime.now() - self._start
        if self._total:
            text = f"{self.title} ({self._cnt}/{self._total} elapsed {delta}) {self._status}"
        else:
            text = f"{self.title} (elapsed {delta}) {self._status}"
        # resize if needed
        return text[:self._width-10]

    def current(self) -> int:
        return self._cnt

    def incr(self) -> None:
        self._cnt += 1

    def update(self, status: str) -> None:
        self._status = status.strip()

Then in my code, I have something like this:

        fmt = BarFormat(bar_title, total=len(bar_items))
        with yaspin(text=fmt).green.arc as bar:
            for line in p.stdout:
                l = line.decode('utf-8').strip()
                if l:
                    # update status of bar only if there's output
                    fmt.update(l)
                if bar_items:
                    if l == bar_items[fmt.current()]:
                        bar.write("> "+l)
                        fmt.incr()
            rc = p.wait()

            # final spinner status
            if rc in ignore_rcs:
                bar.green.ok()
            else:
                bar.red.fail()

This produces something like:

...
> compile code
> <other steps...>
<spinner> Running Some Tool (125/640 elapsed 0:12:34.56789) @I: some info

I noticed that yaspin doesn't handle lines in the text field that overflow the terminal width. I made it so that the width is determined when the class is instanced. It would be super easy, though, to update this to be dynamic.

EDIT: I actually ditched yaspin. I started using rich instead because I decided to update some of my logs to use RichHandler and found out it supports spinners and progress bars. Here's information about that https://rich.readthedocs.io/en/latest/progress.html

Upvotes: 0

aviso
aviso

Reputation: 2841

Here's a working example with the original ask using enlighten. Enlighten will keep the progress bar at the bottom of the terminal and you can print whatever you like through normal methods. See the docs for how to customize.

from subprocess import Popen, PIPE

import enlighten


cmd = 'for i in {1..80}; do sleep 0.2; echo $i; [ $((i % 10)) -eq 0 ] && echo step $((i/10)); done'
lines_in_build_script = [
    'step 1\n',
    'step 2\n',
    'step 3\n',
    'step 4\n',
    'step 5\n',
    'step 6\n',
    'step 7\n',
    'step 8\n',
]

manager = enlighten.get_manager()
pb = manager.counter(total=len(lines_in_build_script), desc='Long Process')

p = Popen(cmd, shell=True, stdout=PIPE, text=True)
pb.refresh()
for line in p.stdout:
    print(line, end='')
    if line in lines_in_build_script:
        pb.update()
rc = p.wait()

Upvotes: 1

Related Questions