CODEWITHSUNDEEP
python
user37292
user37292

Reputation: 334

How is data transferred between two Python scripts

as an example I have two scripts, say script1.py

f = open("output1.txt", "w")
count = 1
for i in range(100):                  
         f.write(str(count) + "\n")
         print(str(count))
         count +=1
f.close

This script prints numbers from 1 to 100 to a file and to standard output.

Then I have a second script, say script2.py

import sys
import time
stdin = sys.stdin
f1 = open("output2.txt", "w")
for line in stdin:
    if len(line)>0:
        print(line.strip())
        time.sleep(0.05)
        f1.write(line.strip() + "\n")

which reads data from standard input and prints them to a file. I added a time.sleep command to ensure the second script consumes data at a far lower rate than they are produced by the first one.

I run the scripts from the command line as

python3 script1.py | python3 script2.py

so redirecting the standard output of the first (so the print() command) to the standard input of the second one.

It works as somehow expected, two files are generated containing numbers from 1 to 100.

I am nevertheless wondering how the data transfer part works, from first to second script.

  1. the first script generates data at a faster rate. Where are these data stored, waiting for the second script to access them?
  2. Is there some sort of buffer that is put in place between the two process? Or what else?
  3. Is Python responsible for this, or the OS?
  4. Is the buffer limited in size? Can it be programmed (e.g. accessed to direct data to another target as well)?

Thanks a bunch

Upvotes: 3

Views: 155

Answers (1)

Ren
Ren

Reputation: 155

it is because of the pipe "|", more info here: https://ss64.com/nt/syntax-redirection.html

commandA  |  commandB     Pipe the output from commandA into commandB

so the prints from your script1 are sent to your script2.

My guess on how it works is that every print is saved in memory as a big string and then sent back (as text) to the second that s why sys.stdin works

Upvotes: 1

Related Questions