CODEWITHSUNDEEP
gcccompiler-constructionx86-64calling-conventionabi
Simon Friese
Simon Friese

Reputation: 111

GCC modified/custom target with a custom calling convention on x86-64

I was wondering if it's possible (in a reasonable amount of time) to modify an existing or to define a new build target for gcc. The goal is to do minor modifications like changing the register in which the result of a function is returned or passing arguments using different registers than by the standard SystemV ABI for x86-64.

For example if I want to change the register for the return value of a function from RAX to R8 or returning a 64-byte struct using YMM0 and YMM1 without writing it myself using x86 asm code.

(Background) I'm working on a small self-written OS so I was wondering if it's possible to not stick to System-V ABI or the one used by Microsoft for Windows. Compatibility with existing libs and stuff is not neccessary as every line of OS code is written by myself. I'm only using gcc, ld and objdump. No gdb, etc...

If this is possible, where do I have to make such modifications in the gcc source or config files? I've cloned the gcc git repo, but couldn't find a starting point for such a modification.

Upvotes: 5

Views: 1184

Answers (1)

rici
rici

Reputation: 241881

This is all a bit theoretical, since I've never actually attempted a complete customisation. But I think it would be a reasonable project.

GCC internals are documented in the GCC internals manual. You probably don't need to read all of it in detail but you should at least try to get an overview of the basic structure. It is all highly configurable.

The calling convention is configured in the "machine description", which in part of the back end. The most important part for the type of changes you are looking at is the description of the stack layout and calling conventions. For example, you can change the way functions return scalar values using the TARGET_FUNCTION_VALUE hook, although you probably will also need to modify other parts of the machine description to correctly handle the register assignment.

I'd suggest starting with a very small modification and thoroughly testing it (at least by verifying that the assembler output is as expected). Then you can implement other modifications, as required. As you develop the changes, keep a collection of test cases; these will be extremely useful to help ensure that changes don't have unexpected interactions with each other.

Upvotes: 5

Related Questions