Serialize list of complex numbers in python for Java

Question

I have this pipeline where i stream data from Python and connect to the stream in a Java applicaton. The data records are matrices of complex numbers. Now I've learned that json.dumps() can't deal with pythons complex type.

For the moment I've converted the complex values to a string, put it in a dictionary like this:

for entry in range(len(data_array)):
    data_as_string = [str(i) for i in data_array[entry]["DATA"].tolist()]
    send({'data': data_array[entry]["DATA"],
          'coords': data_array[entry]["UVW"].tolist()})

and send it to he pipeline. But this requires extensive and expensive custom deserialization in Java, which increases the running time of the pipeline by a lot. Currently I'm doing the deserialization like this:

JSONObject = new JSONOBJECT(string);
try {
            data= jsonObject.getString("data");
            uvw= jsonObject.getString("uvw");
        } catch (JSONException ex) {
            ex.printStackTrace();
        }

And then I'm doing a lot of data.replace(string1, string2) to remove some of the signs added by the serialization and then looping through the matrix to convert every number into a Java Complex type. My Java deserialization code looks the following:

        data = data.replace("(","");
        data = data.replace(")","");
        data = data.replace(""","");
        data = data.replace("],[","¦");
        data = data.replace("[","");
        data = data.replace("]","");

        uvw = uvw.replace("[","");
        uvw = uvw.replace("]","");

        String[] frequencyArrays = data.split("¦");
        Complex[][] tempData = new Complex[48][4];
        for(int i=0;i< frequencyArrays.length;i++){
            String[] complexNumbersOfAFrequency = frequencyArrays[i].split(", ");
            for(int j =0;j



Now my question would be if there is a way to either

1)Deserialize the Dictionary in Java without expensive String manipulations and looping through the matrices for each record or

2)Do a better Job in serializing the data in python so that it can be done better in java

Any hints are appreciated. 

Edit: JSON looks the following

{"data": ["[(1 + 2j), (3 + 4j), ...]","[(5 + 6j), ...]", ..."],

"coords": [1,2,3]}


Edit: For the coordinates I can do the deserialization in Java pretty easily:

uvw = uvw.replace("[","");
uvw = uvw.replace("]","");
String[] coords = uvw.split(",");


And then cast the Strings in coords with Double.parseDouble(), howver for the data string this is way more complicated because the string is full of characters that need to be removed in order to get the actual numbers and to put them in the right place in the Complex[][] I want to cast it to

VGR · Accepted Answer

You are over-using JsonObject.getString, by using it to retrieve non-string data.

Let’s start with the coords property, since it’s a simpler case. [1,2,3] is not a string. It’s an array of numbers. Therefore, you should retrieve it as an array:

JsonArray coords = jsonObject.getJsonArray("coords");

int count = coords.size();
double[] uvw = new double[count];
for (int i = 0; i < count; i++) {
    uvw[i] = coords.getJsonNumber(i).doubleValue();
}

The other property, data, is also an array, but with string elements:

JsonArray data = jsonObject.getJsonArray("data");

int count = data.size();
for (int i = 0; i < count; i++) {
    String complexValuesStr = data.getString(i);
    // ...
}

As for parsing out the complex numbers, I wouldn’t make all those String.replace calls. Instead, you can look for each complex value with a regular expression matcher:

Pattern complexNumberPattern = Pattern.compile(
    "$\s*" +         // opening parenthesis
    "(-?[0-9.]+)" +     // group 1: match real part
    "\s*([-+])\s*" +  // group 2: match sign
    "([0-9.]+)j" +      // group 3: match imaginary part
    "\s*$");         // closing parenthesis
Matcher matcher = complexNumberPattern.matcher("");

JsonArray data = jsonObject.getJsonArray("data");
int count = data.size();

List> allFrequencyValues = new ArrayList<>(count);

for (int i = 0; i < count; i++) {

    String complexValuesStr = data.getString(i);

    List singleFrequencyValues = new ArrayList<>();

    matcher.reset(complexValuesStr);
    while (matcher.find()) {
        double real = Double.parseDouble(matcher.group(1));
        boolean positive = matcher.group(2).equals("+");
        double imaginary = Double.parseDouble(matcher.group(3));

        Complex value = new Complex(real, positive ? imaginary : -imaginary);
        singleFrequencyValues.add(value);
    }

    allFrequencyValues.add(singleFrequencyValues);
}

You should not catch IndexOutOfBoundsException or NumberFormatException. Those indicate the input was invalid. You should not treat invalid input like it’s zero; it means the sender made an error, and you should make sure to let them know it. An exception is a good way to do that.

I have made the assumption that both terms are always present in each complex expression. For instance, 2i would appear as 0 + 2j, not just 2j. And a real number like 5 would appear as 5 + 0j. If that is not a safe assumption, the parsing gets more complicated.

Since you are concerned with performance, I would try the above; if the use of a regular expression makes the program too slow, you can always look for the parentheses and terms yourself, by stepping through the string. It will be more work but may provide a speed increase.

Serialize list of complex numbers in python for Java

Answers (2)

Related Questions