java could not read output of python process until close the input stream

Question

A have some issue while creating a manager for subprocess run. On of such subprocesses is python scripts. I need to write/read/wrap errors of the process in different threads. I have created some simple test:

test.py

import sys

if __name__ == "__main__":
 print ("START TEST SCRIPT")
 print ("SOME STRING")
 for line in sys.stdin:
  print ("DEBUG>>> "+ line)

And create a simpe java test class:

public class Test {

    public static void main(String[] args) {
        ProcessBuilder builder = new ProcessBuilder("python", "test.py");
        builder.environment().put("PYTHONIOENCODING", "UTF-8");
        builder.directory(new File("./test/external_processes/python"));
        try {
            Process environmentProcess = builder.start();
            InputStream out = environmentProcess.getInputStream();
            InputStream err = environmentProcess.getErrorStream();
            OutputStream in = environmentProcess.getOutputStream();

            Thread t1 = new Thread(() -> {
                OutputStreamWriter writer = new OutputStreamWriter(in);
                try {
                    writer.write("new string");
                    writer.flush();
                    writer.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            });

            Thread t2 = new Thread(() -> {
                byte[] buffer = new byte[4096];
                int count;
                try {
                    while ((count = out.read(buffer)) >= 0) {
                        System.out.println("[TASK STREAMER] " + new String(buffer, 0, count));
                    }
                    out.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            });

            Thread t3 = new Thread(() -> {
                byte[] rbuffer = new byte[4096];
                int rcount;
                try {
                    int rno = err.available();
                    while ((rcount = err.read(rbuffer)) >= 0) {
                        System.out.println("[TASK STREAMER ERR] " + rno + " :: " + new String(rbuffer, 0, rcount));
                    }
                    err.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            });

            t1.start();
            t2.start();
            t3.start();

            t1.join();
            t2.join();
            t3.join();

            environmentProcess.waitFor();
            System.out.println("ok!");

            environmentProcess.getInputStream().close();
            System.exit(0);
        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

And everything goes fine

Connected to the target VM, address: '127.0.0.1:50726', transport: 'socket'
[TASK STREAMER] START TEST SCRIPT
SOME STRING
DEBUG>>> new string

Disconnected from the target VM, address: '127.0.0.1:50726', transport: 'socket'
ok!

Process finished with exit code 0

until I decided to not to write anything to output, so I delete Thread t1. After that, a cant reaches the first two lines from python ("START TEST SCRIPT", "START TEST SCRIPT") and execution is stuck.

So if code looks like this:

import java.io.*;

public class Test {

    public static void main(String[] args) {
        ProcessBuilder builder = new ProcessBuilder("python", "test.py");
        builder.environment().put("PYTHONIOENCODING", "UTF-8");
        builder.directory(new File("./test/external_processes/python"));
        try {
            Process environmentProcess = builder.start();
            InputStream out = environmentProcess.getInputStream();
            InputStream err = environmentProcess.getErrorStream();

            Thread t2 = new Thread(() -> {
                byte[] buffer = new byte[4096];
                int count;
                try {
                    while ((count = out.read(buffer)) >= 0) {
                        System.out.println("[TASK STREAMER] " + new String(buffer, 0, count));
                    }
                    out.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            });

            Thread t3 = new Thread(() -> {
                byte[] rbuffer = new byte[4096];
                int rcount;
                try {
                    int rno = err.available();
                    while ((rcount = err.read(rbuffer)) >= 0) {
                        System.out.println("[TASK STREAMER ERR] " + rno + " :: " + new String(rbuffer, 0, rcount));
                    }
                    err.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            });

            t2.start();
            t3.start();

            t2.join();
            t3.join();

            environmentProcess.waitFor();
            System.out.println("ok!");

            environmentProcess.getInputStream().close();
            System.exit(0);
        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

I will reach this:

Connected to the target VM, address: '127.0.0.1:50726', transport: 'socket'

I understand that process won't stop because of python awaiting input, but I at least expect two first strings ("START TEST SCRIPT", "START TEST SCRIPT").

But if I close output stream first, I will reach this strings. Can anybody help me with an understanding of this case, and how to reach subprocess output if it internally has ether input reading and writing to output?

p.s. sorry for my russian grammatic.

Answer 1

If it will be useful to anybody I have found an alternative way without changing client python code.

A has added to builder some additional environment:

builder.environment().put("PYTHONUNBUFFERED","TRUE");

Answer 2

In your python code, you need to flush the print output.

import sys

if __name__ == "__main__":
 print ("START TEST SCRIPT", flush=True)
 print ("SOME STRING", flush=True)
 for line in sys.stdin:
  print ("DEBUG>>> "+ line)