CODEWITHSUNDEEP
pythonbashpython-3.xprocessbenchmarking
Daniel Saad
Daniel Saad

Reputation: 425

Python 3 subprocesses slower than equivalent bash

I'm using a python3 script to automatize some jobs. I need to measure the time of such external jobs. So I decided to use python 3 built-in time() combined with the subprocess module:

with open(in_files[i],'r') as f, open(sol_files[i],'w') as f_sol:
        start = time.time()
        process = subprocess.run(['./'+src_files[i]], stdin = f, stdout=f_sol)
        end = time.time()

The calculated elapsed time by this python snippet is 0.73 seconds

However, the equivalent bash command:

time ./file < input_file > output_file

Is significantly faster: 0.5 seconds

Which could be causing this huge discrepancy? Maybe the context switching with the python interpreter due the redirection usage? Maybe something related to buffering?

A similar code without the redirection usage does not show this behavior:

start = time.time()
process = subprocess.run(['sleep','1'])
end = time.time()

The above code time is elapsed in 1s + negligible time.

Best regards

Upvotes: 1

Views: 255

Answers (1)

Daniel Saad
Daniel Saad

Reputation: 425

It was a stupid mistake.

time.time() does not have a good precision in most systems.

Note that even though the time is always returned as a floating point number, not all systems provide time with a better precision than 1 second. While this function normally returns non-decreasing values, it can return a lower value than a previous call if the system clock has been set back between the two calls. Python 3 Time Module Documentation

perf_counter() or process_time() works just fine. Nothing wrong with subprocesses.

Upvotes: 1

Related Questions