Accessing void pointer values after passing from python cffi to C

Question

I'm using the cffi package in python to test some C code and running into some issues with passing data to C. For context I'm using ABI mode as I can't modify the C source code to work in API mode (code provided below is just an example of the problem).

The particular code I'm testing contains a function that is passed a struct that has a void pointer pointing to another struct which itself has a void pointer pointing to another struct.

test.h

typedef enum State {
    state_1 = 0,
    state_2,
    state_3,
    state_4
} state_t;

typedef struct buffer {
    char* name;
    state_t state;
    void* next;
} buffer_t;

typedef struct buffer_next {
    char* name;
    state_t state;
    void* next;
} buffer_next_t;

typedef struct buffer_next_next {
    char* name;
    state_t state;
    void* next;
} buffer_next_next_t;


extern buffer_t createBuffer();
extern int accessBuffer(buffer_t buffer);

test.c

buffer_t createBuffer(){
    buffer_next_next_t bufferNN;
    buffer_next_t bufferN;
    buffer_t buffer;

    bufferNN.name = "buffer_next_next";
    bufferNN.state = 3;

    bufferN.name = "buffer_next";
    bufferN.state = 2;
    bufferN.next = &bufferNN;

    buffer.name = "buffer";
    buffer.state = 1;
    buffer.next = &bufferN;

    accessBuffer(buffer);

    return buffer;
}

int accessBuffer(buffer_t buffer){
    buffer_next_t *buffer_next = (buffer_next_t*)buffer.next;
    
    buffer_next_next_t *buffer_next_next = (buffer_next_next_t*)buffer_next->next;

    printf("%s, %s, %s\n", buffer.name, buffer_next->name, buffer_next_next->name);

    return 0;
}

test.py

import os
import subprocess
from cffi import FFI

ffi = FFI()

here = os.path.abspath(os.path.dirname(__file__))
header = os.path.join(here, 'test.h')

ffi.cdef(subprocess.Popen([
    'cc', '-E',
    header], stdout=subprocess.PIPE).communicate()[0].decode('UTF-8'))
lib = ffi.dlopen(os.path.join(here, 'test.so'))

value = lib.createBuffer()
print(value)
lib.accessBuffer(value)

Running the above code usually gives a segmentation fault when value is passed to accessBuffer in the python code, as there is no actual string at the addresses of the nested structs for printf to access.

buffer, buffer_next, buffer_next_next
<cdata 'buffer_t' owning 24 bytes>
Segmentation fault

Confirming this in gdb gives:

Breakpoint 1, accessBuffer (buffer=...) at test.c:36
36          printf("%s, %s, %s\n", buffer.name, buffer_next->name, buffer_next_next->name);
(gdb) p buffer
$15 = {name = 0x7ffff77ff01d "buffer", state = state_2, next = 0x7fffffffd860}
(gdb) p ((buffer_next_t*)buffer.next)[0]
$16 = {name = 0x7ffff77ff011 "buffer_next", state = state_3, next = 0x7fffffffd880}
(gdb) p ((buffer_next_next_t*)buffer_next->next)[0]
$17 = {name = 0x7ffff77ff000 "buffer_next_next", state = state_4, next = 0x1}
(gdb) continue
Continuing.
buffer, buffer_next, buffer_next_next
<cdata 'buffer_t' owning 24 bytes>

Breakpoint 1, accessBuffer (buffer=...) at test.c:36
36          printf("%s, %s, %s\n", buffer.name, buffer_next->name, buffer_next_next->name);
(gdb) p buffer
$18 = {name = 0x7ffff77ff01d "buffer", state = state_2, next = 0x7fffffffd860}
(gdb) p ((buffer_next_t*)buffer.next)[0]
$19 = {name = 0x963190 "", state = 8, next = 0x7fffffffd948}
(gdb) p ((buffer_next_next_t*)buffer_next->next)[0]
$20 = {name = 0x1 <error: Cannot access memory at address 0x1>, state = 8, next = 0x0}

When the created buffer is passed purely within the C code there is no problem, but once passed to the python code and then back again to the C code the memory becomes corrupted. My assumption is this likely has something to do with python garbage collection, but ultimately I don't really know what's going on here. Ideally I want the majority of my code to be in python, and currently I'm required to both create and pass the struct purely inside C.

I have tried creating the buffer in python using cffi, but the same issue occurs even when the nested buffers are kept alive and in scope.

In summary, is it possible to create the data structure I want here purely in python first and then pass it to accessBuffer, or if that's not possible, am I able to receive the created buffer from C into python and then back into C without this memory corruption issue occuring?

Answer 1

The variable buffer_next_t bufferN; stops existing after the function returns. The pointer buffer.next = &bufferN; is invalid and can't be accessed after the function returns. – KamilCuk

Based on this comment (thanks again), I realised the mistake I was making here. From this I've created the solution in python:

a = ffi.new("char[20]", b"buffer_next_next")
b = ffi.new("char[20]", b"buffer_next")
c = ffi.new("char[20]", b"buffer")

bufferNN_py = ffi.new("buffer_next_next_t *")
bufferNN_py.name = a
bufferNN_py.state = 3

bufferN_py = ffi.new("buffer_next_t *")
bufferN_py.name = b
bufferN_py.state = 2
bufferN_py.next = bufferNN_py

buffer_py = ffi.new("buffer_t *")
buffer_py.name = c
buffer_py.state = 1
buffer_py.next = bufferN_py

value = lib.createBuffer()

lib.accessBuffer(buffer_py[0])

As the function call is coming from the python code where this data was created, it remains pointed to and the result is as expected.

buffer, buffer_next, buffer_next_next
buffer, buffer_next, buffer_next_next

Confirming in gdb:

Breakpoint 1, accessBuffer (buffer=...) at test.c:33
33          printf("%s, %s, %s\n", buffer.name, buffer_next->name, buffer_next_next->name);
(gdb) p buffer
$1 = {name = 0x7ffff77ff01d "buffer", state = state_2, next = 0x7fffffffda00}
(gdb) p ((buffer_next_t*)buffer.next)[0]
$2 = {name = 0x7ffff77ff011 "buffer_next", state = state_3, next = 0x7fffffffda20}
(gdb) p ((buffer_next_next_t*)buffer_next->next)[0]
$3 = {name = 0x7ffff77ff000 "buffer_next_next", state = state_4, next = 0x44}
(gdb) continue
Continuing.
buffer, buffer_next, buffer_next_next

Breakpoint 1, accessBuffer (buffer=...) at test.c:33
33          printf("%s, %s, %s\n", buffer.name, buffer_next->name, buffer_next_next->name);
(gdb) p buffer
$4 = {name = 0xa967d0 "buffer", state = state_2, next = 0xa3ab30}
(gdb) p ((buffer_next_t*)buffer.next)[0]
$5 = {name = 0x9e8220 "buffer_next", state = state_3, next = 0xb35620}
(gdb) p ((buffer_next_next_t*)buffer_next->next)[0]
$6 = {name = 0xa59d40 "buffer_next_next", state = state_4, next = 0x0}