Failed to decode data using encoding 'utf-8' in dart

Question

I am decoding a video file using AES decryption(ECB mode). I have implemented the below code, but get an exception:

FileSystemException (FileSystemException: Failed to decode data using encoding 'utf-8', path = '/data/user/0/bhaveshparakh.acadflip/app_flutter/sample1.ehb')

This is the code i implemented:

var dir =
                                    await getApplicationDocumentsDirectory();

                                Dio dio = Dio();
                                dio.download(videourl, '${dir.path}/sample1.ehb',
                                    onReceiveProgress:
                                        (actualbytes, totalbytes) {
                                  var percentage =
                                      actualbytes / totalbytes * 100;
                                  setState(() {
                                    downloadMessage =
                                        'Downloading ${percentage.floor()} %';
                                  });
                                });
decryptFile('${dir.path}/sample1.ehb');

Future<String> decryptFile(filePath) async{ 
     //'filePath' contains the encrypted video file.
    var encodedKey = info;
    var encryptedBase64EncodedString = await new File(filePath).readAsString(encoding:utf8);
    var decoded = base64.decode(encryptedBase64EncodedString);
    final key1 = enc.Key.fromBase64(encodedKey);
    final encrypter = enc.Encrypter(enc.AES(key1, mode: enc.AESMode.ecb));
    final decrypted = encrypter.decryptBytes(enc.Encrypted(decoded));
    final filename = '${p.basenameWithoutExtension(filePath)}.mp4';
    final directoryName=p.dirname(filePath);
    final newFilePath=p.join(directoryName, filename);
    var newFile = new File(newFilePath);
    await newFile.writeAsBytes(decrypted);
    print("Successfull");
    return newFilePath;
  }

Though its been encrypted the file using utf8 in JAVA, i am getting this error. Is there something i am doing wrong?

This is how they have encrypted the file in JAVA

public static void listFilesForFolder(final File folder) {
        for (final File fileEntry : folder.listFiles()) {
            if (fileEntry.isDirectory()) {
                listFilesForFolder(fileEntry);
            } else {
                System.out.println(fileEntry.getName());
                try {
                    Cipher desCipher;

                    // Create the cipher
                    desCipher = Cipher.getInstance("AES");

                    byte[] key = encodekey.getBytes("UTF-8");
                    MessageDigest sha = MessageDigest.getInstance("SHA-1");
                    key = sha.digest(key);
                    key = Arrays.copyOf(key, 16); // use only first 128 bit

                    SecretKeySpec secretKeySpec = new SecretKeySpec(key, "AES");
                

                    desCipher.init(Cipher.ENCRYPT_MODE, secretKeySpec);

                    byte[] text = readVideo(fileEntry.getName());
            
                    byte[] textEncrypted = desCipher.doFinal(text);
                    writeByte(textEncrypted, fileEntry.getName());
                    
                } catch (Exception e) {
                    
                    e.printStackTrace();
                }

            }
        }
        System.out.println("Done!!!");
    }

    public static byte[] readVideo(String name) {

        byte[] bytes = null;
        try {
            FileInputStream fis = new FileInputStream(new File("/home/raghib/java/source/" + name));
            ByteArrayOutputStream bos = new ByteArrayOutputStream();
            byte[] b = new byte[1024];

            for (int readNum; (readNum = fis.read(b)) != -1;) {
                bos.write(b, 0, readNum);
            }

            bytes = bos.toByteArray();
            fis.close();

        } catch (Exception e) {
            e.printStackTrace();
        }

        return bytes;
    }
    public static void writeByte(byte[] bytearray, String filename) {
        try {
            String replacedStr1 = filename.replaceAll(".mp4", ".ehb");
            String replacedStr = replacedStr1.replaceAll(".pdf", ".ehbp");
            FileOutputStream fileoutputstream = new FileOutputStream(
                    new File("/home/raghib/java/destination/" + replacedStr), true);
            fileoutputstream.write(bytearray);
            fileoutputstream.close();
        } catch (Exception ex) {
            ex.printStackTrace();
        }

    }

Answer 1

There is no such thing as encrypting a file using UTF-8. UTF-8 is an encoding, and it can only encode valid Unicode codepoints. It's not an encoding of arbitrary data. But that's not what you're doing. You're expecting Base64:

var encryptedBase64EncodedString = await new File(filePath).readAsString(encoding:utf8);
var decoded = base64.decode(encryptedBase64EncodedString);

Base64 is an encoding that can transform any data into 7-bit ASCII, and Base64 can itself be encoded as UTF-8.

But your Java code doesn't write Base64. It writes raw, unencoded bytes. I don't know why this variable is called "textEncrypted." It's not text.

writeByte(textEncrypted, fileEntry.getName());

Except I wouldn't expect that to work. writeByte() should write a single byte, not an array of bytes. (writeBytes() accepts an array of bytes.)

Given these problems, it is also very possible that the key encoding is also incorrect. You don't provide enough information to fully evaluate that, but the Java and Dart key handling look completely different.

Encryption algorithms have to be implemented exactly, step-by-step identically on both sides. The algorithm you're using here is custom and highly ad hoc. In order to implement your decryptor, you need to go through the Java and implement each step identically in Dart.

As a separate issue, for encrypting video this is extremely insecure. ECB is a terrible tool for encrypting structured data like video. It leaks way too much information about the contents.