given this "encryption" function, how can I reverse it and write a decryption function?

Question

TL;DR - 1st code just shows what happens before it actually starts doing its thing.Look at the 2nd code, this is what my ASM does to encrypt a string. Look at the 3rd box, it shows the result. Ekey = h. (the code overall is made of C++ and inline assembly)

How can I reverse it back again so it goes back to it's "parent" (original state, before encryption).

---

So I have gotten an assembly code, which encrypts the given string, (body is written in C++, simple cin and couts + for loop to loop each character, nothing serious, but the question applies to assembly)

This is what happens before it goes into encryption routine: (different function)

push   eax              // Push EAX(Ekey) value to the stack
push   ecx              // Push ECX(character to be encrypted) value to the stack

movzx  ecx, temp_char   // Copies the contents of the temp_char to the ECX register and zero extends the value.
lea    eax, EKey        // Load the address of EKey, into EAX
call   encrypt          // pushes the return address onto the stack and transfers control to a calling label(In this case, it's encrypt). 
mov    temp_char, al    // Copy AL into temp_char variable.

pop    ecx              // Get value from the top of the stack into ECX.
pop    eax              // EAX, gets value from the top of the stack.

My encryption routine looks like this: Just to clarify, comments are mine. There's a high chance that they are correct, or some of them might be wrong.(Sorry, I'm still learning)

        push  ebp                      // Save the old base pointer value.                           
        mov   ebp, esp                 // esp; Set the new base pointer value.                       
        sub   esp, 12                  // Make room for 3 registers          
        push ebx                       // Push EBX value to the stack
        push edx                       // Push EDX value to the stack
        push ecx                       // Push ECX(character to be encrypted) value to the stack
        movzx edx, byte ptr[eax]       // Taking first byte from EAX(Ekey), zero-extending it with 0's , moving that into EDX
        and edx, 0x43                  // Starting from the most significant bit. EDX will be set to 0x00(zeroed out) if 2nd, 7th and 8th bits are 0's. Otherwise if any bits are 1 within the mask, the resulting value will be non-zero

        cmp edx, 0x00                  // See if EDX is equal to 0.
        jnz x16                        // If EDX != 0 go to X16.
        mov edx, 0x07                  // If EDX will be equal to 0, set EDX value to 0x07..
        x16:        inc dl             // Add one to the 'DL' register. Character variable moves up // DL is 1st byte of EDX.
        mov dword ptr[eax], edx        // dword 4 bytes, EDX value into EAX(Ekey) and zero extend it to 4 bytes.
        pop ebx                        // Getting value from the top of the stack, and storing it in the EBX.
        y16 : dec ebx                  // Decrement EBX by 1.
        dec edx                        // Decrement EDX by 1.
        jnz y16                        // If EDX != 0 go to Y16.
        not bl                         // Reverse contents of BL. Such as., (Before 0101, after 1010.) <- Example. So given input will be revesed for encryption purposes. BL, is a 1st byte of EBX.

        pop edx                        // Restores original value of EDX
        mov eax, ebx                   // Move contents from EBX register into EAX(Ekey) register.
        pop ebx                        // Get value from the top of the stack, and then store it in EBX. EAX(Ekey) => EBX.              
        mov  esp, ebp                  // Deallocate local variables                                 
        pop  ebp                       // Restore the caller's base pointer value                    
        ret                            // Gets address from the top of the stack(In this case it's EBX) and returns the value as result. Returning the final result.

This is what it does with "soon" string

 Date: 04/07/2018  Time: 13:53:36

Original string =  soon Hex = 73  6f  6f  6e  

Encrypted string = ÍÒÓÕ Hex = cd  d2  d3  d5  

I've spent unaccounted hours on this, to get it "reversed" but nothing major came across my mind. The final result, was that I am short of one keyboard. I've already looked up on the internet(this includes stackoverflow) on how to accomplish such goal although I haven't succeeded. So this is my last resort here.

How do I reverse those instructions to get it back to the original string?

Answer 1

Here is a C version of your program; I think it makes it pretty self evident how it works:

#include <libc.h>
int ec(unsigned char *s, int c) {
    unsigned char y;
    y = *s & 0x43;
    if (y == 0) {
        y = 7;
    }
/*-
  NB: y is in { 7,1,2,3,64,65,66,67 }
*/
    *s = ++y;
    c -= y;
    return ~c & 0xff;
}

void estr(char *s) {
    unsigned char k;
    k = 'h';
    while (*s) {
        *s = ec(&k, *s);
        s++;
    }
}

int main() {
    char p[20];
    strcpy(p, "Hello, world!");
    printf("%s\n", p);
    estr(p);
    printf("%s\n", p);
    estr(p);
    printf("%s\n", p);
}

There are two things going on here; the first is the key sequence. The key starts from a given seed value (‘H’ = 0x48). Which generates the sequence: 0x41,0x42,0x43,0x44,0x41,.... by the operation key = (key AND 0x43)+1; note the weird exception for 0 in the code.

This sequence is applied to each character by the encryption function: c = NOT (c - key). This function is its own inverse, thus if you apply it to plain text, it yields encrypted text and visa versa.

I hope that helps.