CODEWITHSUNDEEP
linuxx86
sudo rm -rf slash
sudo rm -rf slash

Reputation: 1254

x86 segfault in "call" to function

I am working on a toy compiler. I used to allocate all memory with malloc, but since I never call free, I think it will be sufficient (and faster) to allocate a GB or so on the stack and then slowly use that buffer.

But... now I am segfaulting before anything interesting happens. It happens on about 30% of my test cases (all test cases are the same in this section tho). Pasted from GDB:

(gdb) disas
Dump of assembler code for function main:
   0x0000000000400bf1 <+0>:    push   rbp
   0x0000000000400bf2 <+1>:    mov    rbp,rsp
   0x0000000000400bf5 <+4>:    mov    QWORD PTR [rip+0x2014a4],rsp        # 0x6020a0
   0x0000000000400bfc <+11>:    sub    rsp,0x7735940
   0x0000000000400c03 <+18>:    sub    rsp,0x7735940
   0x0000000000400c0a <+25>:    sub    rsp,0x7735940
   0x0000000000400c11 <+32>:    sub    rsp,0x7735940
=> 0x0000000000400c18 <+39>:    call   0x400fec <new_Main>
   0x0000000000400c1d <+44>:    mov    r15,rax
   0x0000000000400c20 <+47>:    mov    rax,r15
   0x0000000000400c23 <+50>:    add    rax,0x20
   0x0000000000400c27 <+54>:    mov    rax,QWORD PTR [rax]
   0x0000000000400c2a <+57>:    add    rax,0x48
   0x0000000000400c2e <+61>:    mov    rax,QWORD PTR [rax]
   0x0000000000400c31 <+64>:    call   rax
   0x0000000000400c33 <+66>:    mov    rax,0x0
   0x0000000000400c3a <+73>:    mov    rsp,rbp
   0x0000000000400c3d <+76>:    pop    rbp
   0x0000000000400c3e <+77>:    ret

I originally did one big "sub rsp, 0x..." and I thought breaking it up a bit would help (it didn't -- the program crashes at call either way). The total should be 500MB in this case.

What really confuses me is why it fails on "call <>" instead of one of the subs. And why it only fails some of the time rather than always or never.

Disclosure: this is a school project, but asking for help with general issues regarding x86 is not against any rules.

Update: based on @swift's comment, I set ulimit -s unlimited... and it now segfaults randomly? It seems random. It's not coming close to using the whole 500 MB buffer tho. It only allocates about 400 bytes total.

Upvotes: 2

Views: 820

Answers (1)

Sami Kuhmonen
Sami Kuhmonen

Reputation: 31193

Subtracting something from RSP won’t cause any issues since nothing uses it. It’s just a register with a value, it doesn’t allocate anything. But when you use CALL then memory pointed by RSP is accessed and issues may happen. The stack usually isn’t very big so to your question “is there any reason you can’t take a GB of memory from the stack” the answer is “because the stack doesn’t have that much space to be used.”

As for being faster to allocate a big buffer in the stack isn’t really a thing. Allocating and releasing a single big block of memory isn’t slower in the heap. Having lots of allocations and releases in heap is worse than in the stack. So there’s not much point in this case to do it in the stack.

Upvotes: 1

Related Questions