CODEWITHSUNDEEP
encryptioncryptographybytereverse-engineeringxor
user1704029
user1704029

Reputation: 275

Reverse engineering a file encryption (most likely XOR)

Im trying to reverse engineer a file format which is encrypted. Most likely it uses an XOR encryption. I can create encrypted files with known plaintext, which I analyzed:

enc     71 8d 7e 84 29 20 b8 cb 6c ed bb 8a 62 a1 
dec     74 68 69 73 20 69 73 20 61 20 74 65 73 74 
xor     05 e5 17 f7 09 49 cb eb 0d cd cf ef 11 d5 
txt     t  h  i  s     i  s     a     t  e  s  t  

enc     61 ad be 84 29 20 b8 cb 6c ed bb 8a 62 a1 
dec     64 68 69 73 20 69 73 20 61 20 74 65 73 74 
xor     05 c5 d7 f7 09 49 cb eb 0d cd cf ef 11 d5 
txt     d  h  i  s     i  s     a     t  e  s  t 

enc     62 a5 ae a4 e9 a0 b8 cb 6c ed bb 8a 62 a1 
dec     67 68 69 73 20 69 73 20 61 20 74 65 73 74 
xor     05 cd c7 d7 c9 c9 cb eb 0d cd cf ef 11 d5
txt     g  h  i  s     i  s     a     t  e  s  t

It is obvious that the original text is part of the encryption. The first byte of the key is always 05. The second byte of the key can be calculated like this:

(enc1 + dec1) OR xor1

The rather low entropy of the key implies a similar rule for the other key-bytes.

Any ideas?

Upvotes: 1

Views: 2137

Answers (1)

dna
dna

Reputation: 1075

You almost got it!

The key's byte at the m position is given by :

km = [(en + dn) ^ kn] | secret

where : 

en is the previous encrypted byte
dn is the previous plain text byte
kn is the previous key byte (k0 = 5)
secret is an arbitrary number starting at 5 and incremented by 2 every two turns
^ is the xor operator
| is the or operator

A simple C# key generator :

namespace Sample.CustomEncrypt {
    using System.Collections.Generic;
    using System.Text;

    class Program {
        static void Main() {
            var key1 = GenerateKey("this is a test");
            var key2 = GenerateKey("dhis is a test");
            var key3 = GenerateKey("ghis is a test");
        }

        public static byte[] GenerateKey(string input) {
            var plain = Encoding.UTF8.GetBytes(input);
            var secret = 5;
            var key = new List<byte> {
                0x05
            };

            for (var i = 0; i < plain.Length - 1; i++) {
                var dn = plain[i];
                var kn = key[i];
                var en = (byte)(dn ^ kn);
                var km = (byte)(((dn + en) ^ kn) | secret);

                key.Add(km);

                if (i % 2 == 0) {
                    secret += 2;
                }
            }

            return key.ToArray();
        }
    }
}

PS: As pointed out by Eugene you should post on Reverse Engineering or Cryptography next time.

Upvotes: 2

Related Questions