CODEWITHSUNDEEP
pythontqdm
Max Ehrlich
Max Ehrlich

Reputation: 2525

Read TQDM Output From File

This is mostly a general python question, I am working in a distributed environment so my workers write their stdout and stderr to files on a shared filesystem (I can't change this). I want my launcher script to read one of those files and print it out realtime to roughly track the progress of the workers, each of which outputs progress bars using tqdm. I was happy to see that running tail -f of the files correctly outputs the progress bars e.g. the bars update on a single line just like they would if I was running tqdm locally. However when I try to wrap that with python, using the following function:

def follow(job : submitit.Job) -> Iterator[str]:
    with open(job.paths.stdout) as fo:
        with open(job.paths.stderr) as fe:
            while True:
                lo = fo.readline()
                    
                if lo:
                    yield lo         

                le = fe.readline()
                if le:
                    yield le

                if job.state != 'RUNNING' and not (le or lo):
                    break
                else:
                    time.sleep(1)

loglines = follow(job)
for line in loglines:
    print(line, end='', flush=True)

it doesn't do this, e.g. I get a newline and the progress bar updates by printing new lines, it doesn't seem to handle the \x1b[A character that tqdm is writing.

Is there a way to do this purely in python?

Upvotes: 0

Views: 758

Answers (1)

Max Ehrlich
Max Ehrlich

Reputation: 2525

I was able to solve this by using read binary mode, using read to read to the end of the file, then decoding the resulting bytes as ascii instead of the python 3 default of UTF-8. The updated working follow function is as follows:

def follow(job : Job) -> Iterator[str]:
    with open(job.paths.stdout, 'rb') as fo:
        with open(job.paths.stderr, 'rb') as fe:
            while True:
                lo = fo.read().decode('ascii')
                    
                if lo:
                    yield lo   

                le = fe.read().decode('ascii')
                if le:
                    yield le

                if job.state != 'RUNNING' and not le and not lo:
                    break
                else:
                    time.sleep(0.1)

Upvotes: 1

Related Questions