Receiving python json through sockets with java

Question

I've been following multiple tutorials to connect a java code to python through sockets.

Sending from java to python works great, with json array. However, I can't seem to receive things with java. I don't really understand how the listening should be done. Now I just listen in a 15 second while loop (the python should send as soon as it receives an input), but I feel like I am doing something substantially wrong. Maybe somebody has an idea?

client.py:

import socket
import sys
import numpy as np
import json

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Bind the socket to the port
server_address = ('localhost', 10004)
print >>sys.stderr, 'starting up on %s port %s' % server_address
sock.bind(server_address)

# Listen for incoming connections
sock.listen(1)

def mysum(x):
    return np.sum(x)

while True:
    # Wait for a connection
    print >>sys.stderr, 'waiting for a connection'
    connection, client_address = sock.accept()
    infile = sock.makefile();

    try:
        print >>sys.stderr, 'connection from', client_address

        # Receive the data in small chunks and retransmit it
        data = b''
        while True:
             new_data = connection.recv(16)
             if new_data:
                 # connection.sendall(data)
                 data += new_data
             else:
                 data += new_data[:]
                 print >>sys.stderr, 'no more data from', client_address
                 break
        data= data.strip('\n');
        print("data well received!: ")
        print(data,)
        print(np.array(json.loads(data)));

        #send a new array back

        sendArray =  np.array( [ (1.5,2,3), (4,5,6) ] );
        print("Preparing to send this:");
        print(sendArray);
        connection.send(json.dumps(sendArray.tolist()));

    except Exception as e:
        print(e)
        connection.close()
        print("closed");
    finally:
        # Clean up the connection
        connection.close()
        print("closed");

server.java:

import java.io.*;
import java.net.Socket;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;

import org.json.*;

import java.net.ServerSocket;

public class SocketTest {

    public static void main(String[] args) throws IOException {


        String hostName = "localhost";
        int portNumber = 10004;

        try (

                //open a socket
                Socket clientSocket = new Socket(hostName, portNumber);

                BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
                PrintWriter out = new PrintWriter(clientSocket.getOutputStream(), true);


        ) {

            System.out.println("Connected");
            Double[][] test2 = new Double[5][2];
            test2[1][1] = 0.1;
            test2[1][0] = 0.2;
            test2[2][1] = 0.2;
            test2[2][0] = 0.2;
            test2[3][1] = 0.1;
            test2[3][0] = 0.2;
            test2[4][1] = 0.2;
            test2[4][0] = 0.2;
            test2[0][1] = 0.2;
            test2[0][0] = 0.2;


            System.out.println("A");
            out.println(new JSONArray(test2).toString());
            System.out.println("B");


            long t = System.currentTimeMillis();
            long end = t + 15000;


            while (System.currentTimeMillis() < end) {
                String response;
                while ((response = in.readLine()) != null) {
                    System.out.println("receiving");
                    System.out.println( response );

                }


            }


            //listen for input continuously? 


            //clientSocket.close();
        } catch (JSONException e) {
            e.printStackTrace();
        }


    }


}

The output from python is:

data well received!: 
('[[0.2,0.2],[0.2,0.1],[0.2,0.2],[0.2,0.1],[0.2,0.2]]',)
[[ 0.2  0.2]
 [ 0.2  0.1]
 [ 0.2  0.2]
 [ 0.2  0.1]
 [ 0.2  0.2]]
Preparing to send this:
[[ 1.5  2.   3. ]
 [ 4.   5.   6. ]]
closed
waiting for a connection
connection from ('127.0.0.1', 40074)

Output from java:

A
B

The problem is that sendArray = np.array( [ (1.5,2,3), (4,5,6) ] ); is never received by java. I feel like I am missing something simple to make it listen... Thanks for any help.

Answer 1

You code has multiple issues.

1. Client.py and Server.java both waiting to receive data, which causes deadlock.

Client.py is blocked at while True, keeps waiting for new data.
Server.java has sent the JSONArray(test2).toString() whose length is 51. Then keeps waiting at in.readLine().

See How to identify end of InputStream in java. The same idea works in python. It's better to know how many bytes you want to read.

Change: Because JSONArray(test2).toString() has length 51, so you replace

    while True:
         new_data = connection.recv(16)
         if new_data:
             # connection.sendall(data)
             data += new_data
         else:
             data += new_data[:]
             print >>sys.stderr, 'no more data from', client_address
             break

With

 data = connection.recv(51)

2. Server.java calls in.readLine(). But Client.py never sending the '\n' and closed the socket too soon. It causes in.readLine() throw exception.

Change: send '\n' so Server.java can successfully read a line.

    connection.send(json.dumps(sendArray.tolist()));
    connection.send("\n");

3. Client.py close the socket, which causes the endless in.readLine() throws "Connection reset" exception.

Changes: Make sure both sides don't send/receive data anymore, then close the socket together.

Code after changes of 1&2. (It requires more efforts to solve 3 and it's not the point of this question):

public class Server {

  public static void main(String[] args) throws IOException {


    String hostName = "localhost";
    int portNumber = 10004;

    try (

        //open a socket
        Socket clientSocket = new Socket(hostName, portNumber);

        BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
        PrintWriter out = new PrintWriter(clientSocket.getOutputStream(), true);


    ) {

      System.out.println("Connected");
      Double[][] test2 = new Double[5][2];
      test2[1][1] = 0.1;
      test2[1][0] = 0.2;
      test2[2][1] = 0.2;
      test2[2][0] = 0.2;
      test2[3][1] = 0.1;
      test2[3][0] = 0.2;
      test2[4][1] = 0.2;
      test2[4][0] = 0.2;
      test2[0][1] = 0.2;
      test2[0][0] = 0.2;


      System.out.println("A");
      out.println(new JSONArray(test2).toString());
      System.out.println("B");


      long t = System.currentTimeMillis();
      long end = t + 15000;


      while (System.currentTimeMillis() < end) {
        String response;
        while ((response = in.readLine()) != null) {
          System.out.println("receiving");
          System.out.println( response );

        }
      }
      //listen for input continuously?
      //clientSocket.close();
    } catch (JSONException e) {
      e.printStackTrace();
    }
  }
}

Client.py output:

starting up on localhost port 10004
waiting for a connection
connection from ('127.0.0.1', 53388)
data well received!: 
('[[0.2,0.2],[0.2,0.1],[0.2,0.2],[0.2,0.1],[0.2,0.2]]',)
[[ 0.2  0.2]
 [ 0.2  0.1]
 [ 0.2  0.2]
 [ 0.2  0.1]
 [ 0.2  0.2]]
Preparing to send this:
[[ 1.5  2.   3. ]
 [ 4.   5.   6. ]]
closed

Server.java output:

Connected
A
B
receiving
[[1.5, 2.0, 3.0], [4.0, 5.0, 6.0]]
Exception in thread "main" java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:209)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)

Answer 2

This is happening because your java code is blocking. See how B is not printed to your logs? That is because it does't execute since this part is waiting for a flush command: out.println(new JSONArray(test2).toString());. What you need to do is out.flush(); so it goes on.