CODEWITHSUNDEEP
assemblyembeddedcalling-convention
Fredrik Möller
Fredrik Möller

Reputation: 116

Automatically detecting violations of assembly calling convention

Is there a way for the assembler (or a static analyzer) to warn if the hand-coded assembly code contains violations of the platform's assembly calling convention?

The platform I'm using is ARMv7A with the GNU GAS assembler. The reason for the question is a bug I wrote where my function did not push/pop the required registers (r4-r11 on ARM) upun entry/exit. The registers were trashed, causing the caller to crash (thankfully, the automated tests detectected the bug). Simplified program:

my_function:
    mov     r4, #42  @Trash register r4 in violation of calling convention
    bx      lr       @Return from function

caller:
    ...
    mov r4, #4        @Initialise register r4, to be used later
    bl my_function    @Call my_function with no arguments
    mov r0, r4        @Set argument r0 as r4 (== 42, but should be 4)
    bl other_function @Call other_function with (the now trashed) argument r0

ARM calling convention: http://infocenter.arm.com/help/topic/com.arm.doc.ihi0042d/IHI0042D_aapcs.pdf

Upvotes: 4

Views: 486

Answers (1)

Alexey Frunze
Alexey Frunze

Reputation: 62068

I don't think there's such a tool. It is expected that assembly programmers know very well what they're doing.

However, if you're really after it, you can develop such a tool.

In it you'd need to parse the assembly source code and recognize instructions modifying the registers that have to be preserved by a subroutine. Ditto for the instructions saving/restoring those registers.

You will also need to figure out where subroutines begin and end. The beginning can be recognized by noting that the label, designating the entry point into the subroutine, is defined as public/global/whatever, basically, that it is supposed to visible from outside by the linker. Another possible heuristic is seeing that a label is used in call instructions, or whatever the equivalent of it is. Likewise you can determine subroutine ends, by noting the instructions or sequences of them that perform a return.

There may be cases where it's hard to figure out the beginnings and ends or whether a register is being truly trashed or preserved. Some of them can be resolved with extra heuristics. The rest can by default result in warnings. It's probably better to have false negatives, false alarms, then the other way around. If the code is small, figuring out whether the warning is meaningful or not should be easy.

For this task you should consider some scripting or scriptish languages that can handle well strings, support regular expressions and support "standard" containers and algorithms working with them (searching/sorting/etc). Perl and Python could do the job well. I'd not advise doing it in C or C++ because you'd need to rewrite and throw away numerous small pieces in the development process. Compiling is an extra hurdle, however small it may be. Debugging low-level code or templates isn't fun.

Upvotes: 0

Related Questions