CODEWITHSUNDEEP
cvisual-studioassemblycompilation64-bit
Bastien Durel
Bastien Durel

Reputation: 613

visual studio 2013 truncates returned pointer value in x64

I try to compile wingraphviz for x64 (it's an old, unmaintained project), and ran into a very strange problem :

There's a call to getDefaultFont() that looks like this :

const char* def = getDefaultFont();
Deffontname = late_nnstring(g->proto->n,N_fontname,def);

(original code did the call inside function call, but I extracted it for understanding)

the getDefaultFont function is very simple, and returns a string litteral based on current charset :

const char * getDefaultFont() {
    switch(DOT_CODEPAGE) {

            case CP_KOREAN:
            return CP_949_DEFAULTFONT;
            break;
[...]

            default:
            return DEFAULT_FONTNAME;
            break;
    }
}

with DEFAULT_FONTNAME & others defined in a header file :

#define     DEFAULT_FONTNAME    "Times New Roman"

I changed the return to { const char* r = DEFAULT_FONTNAME; return r; } to see the value while debugging: r is correct at return instruction.

But when the debugger returns to caller function, def points to invalid memory.

I ran the debugger in assembly mode, and see that :

    const char* def = getDefaultFont();
000007FEDA1244FE  call        getDefaultFont (07FEDA1291A0h)  
000007FEDA124503  cdqe  
000007FEDA124505  mov         qword ptr [def],rax

after the call instruction, RAX contains the correct value, a pointer to .data : RAX = 000007FEDA0C9A20 but the next instruction, cqde "Convert dword (eax) to qword (rax)." destroy the 4 higher bytes, and now RAX = FFFFFFFFDA0C9A20. Then the third stores the truncated value on stack.

After that, late_nnstring() tries to de-reference the corrupted pointer and crashes...

Do you know why VS inserts this cqde instruction ? All theses functions are in .c files under the same project.

I've implemented a workaround, using strdup() to return low-memory addresses, but it's not safe (maybe heap can use memory after 4G?) (and there my be some other cases I did not find while testing that will crash when using the library)

I published the files here : https://gitlab.com/data-public/wingraphviz

especially :

Upvotes: 1

Views: 176

Answers (1)

Soonts
Soonts

Reputation: 21946

Your links require some account I don’t have.

You likely failed to include the header declaring that function, or messed up with headers order. Here’s more info why C compiler inserts cdqe.

P.S. Great example why you should read, and fix, compiler warnings.

Update: If you have circular dependency problem and can’t just include utils.h, a quick workaround is declare const char * getDefaultFont(); in emit.c before you call that function.

Upvotes: 3

Related Questions