Python generated zlib + base64 text getting incorrect header check exception in Java

Question

I am required to write a consumer in Java from Kafka streams, publisher app is written by a third party in python. When I use base64 decoding and then zlip uncompress in java, I get an incorrect header check exception.

My Task is: Convert this compressed base64 + zlib data into a text file that is readable.

Publisher's code in python:

 # read in the file content
   inputf=open(file, 'rb')
   input=inputf.read()
   inputf.close()
   # using zlib.compress(s) method
   compressed = zlib.compress(input)
   # encoding with base64 encoding
   encoding_type='base64'
   enc_data=encode(compressed,encoding_type)
   enc_data_utf8=enc_data.decode("utf-8")
#   enc_data=enc_data_no_no_newline                      ####[0:86000]       #   trim
   event_etl_event[filename+"_content"]=enc_data_utf8
   event_etl_event[filename+"_compressed_format"]="zlib+uuencode"
    enter code here

Consumer Code in java

public void processData(){

   inputStr = event.getEventEtlEvent().getAllProblemsTxtContent();
        
        
        System.out.println("Before Base64 decoding: \n" + inputStr);
        
         Path path0 = Paths.get("AllProblems_Before_base64_decoding.txt");
         Files.write(path0, inputStr.getBytes());
          
        Base64 base64 = new Base64();
        String decodedString = new String(base64.decode(inputStr.getBytes()));
        
        System.out.println("After Base64 decode: \n" + decodedString);
        
         Path path1 = Paths.get("AllProblems_After_base64_decoding.txt");
         Files.write(path1, decodedString.getBytes());
          
        System.out.println("now zlib 64 decodingString .........\n\n\n");
        
        byte[] output = ZLibUtils.decompress(decodedString.getBytes());  

        System.out.println("After Zlib Decompress: "+ output);

        
    } catch (JsonParseException e) {
        e.printStackTrace();
    } catch (JsonMappingException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    } /*catch (InvalidDataException e) {
        e.printStackTrace();
    } catch (DataFormatException e) {
        e.printStackTrace();
    }*/catch (Exception e) {
        e.printStackTrace();
    }

ZLibUtils.java

  public static byte[] decompress(byte[] data) throws DataFormatException {  
            byte[] output = new byte[0];  
    
            Inflater decompresser = new Inflater();  
            decompresser.reset();  
            decompresser.setInput(data);  
    
            ByteArrayOutputStream o = new ByteArrayOutputStream(data.length);  
           // try {  
                byte[] buf = new byte[data.length]; 
                byte[] a = new byte[data.length];
                while (!decompresser.finished()) {  
                    int i = decompresser.inflate(buf);  
                    o.write(buf, 0, i);  
                }  
                output = o.toByteArray();  
                /* } catch (Exception e) {  
                output = data;  
                e.printStackTrace();  
                //FIXME: in later code
                System.exit(0);
            } finally {  
                try {  
                    o.close();  
                } catch (IOException e) {  
                    e.printStackTrace();  
                }  
            }  */
    
            decompresser.end();  
            return output;  
        }  

Now when I run my program, i get the following exception:

    java.util.zip.DataFormatException: incorrect header check
        at java.util.zip.Inflater.inflateBytes(Native Method)
        at java.util.zip.Inflater.inflate(Inflater.java:259)
        at java.util.zip.Inflater.inflate(Inflater.java:280)
        at com.exmple.util.ZLibUtils.decompress(ZLibUtils.java:84

)

Looking forward to hearing from you soon.

Answer 1

The problem is that you convert the base64 decoded data from a byte array into a string and back into a byte array. For most encodings this is not a no-op. That means, for most encodings and most byte arrays,

byte[] decoded = { (byte) 0x9b, 1, 2, 3 };
String decodedString = new String(decoded);
byte[] processed = decodedString.getBytes();

the contents of processed will be different from the contents of decoded.

The solution is to not treat the base64 decoded data as string and instead of that work with the byte data directly:

    Base64 base64 = new Base64();
    byte[] decoded = base64.decode(inputStr.getBytes());
    
    Path path1 = Paths.get("AllProblems_After_base64_decoding.txt");
    Files.write(path1, decoded);
      
    System.out.println("now zlib 64 decodingString .........\n\n\n");
    
    byte[] output = ZLibUtils.decompress(decoded);