CODEWITHSUNDEEP
cpointersipcshared-memorymmap
vassago
vassago

Reputation: 45

Pointers within shared memory

I'm using shared memory (shm_open/mmap) to keep track of some state. In my shared memory I have the struct:

typedef struct fim_t {
    uint64_t num_procs;
    uint64_t num_numa;
    int64_t *numa_nodes[MAX_FIM_NUMA];
    int64_t procs[MAX_FIM_PROC];
}fim_t;

What I want to do is load process IDs in the procs array and then have the numa_nodes array point to procs array values so I can manipulate the value in one spot and have it change across all the references. My understanding is that setting the numa_nodes references to addresses of the procs array should not be a memory access violation because their addresses are both entirely within the shared memory segment. However I get a seg fault when I try to access the value which tells me that my previous statement must be false.

Here is example code:

int main(){
    int fd;
    int init_flag = 0;
    if((fd = shm_open("fim", O_RDWR | O_CREAT | O_EXCL, S_IRWXU)) > 0){
        printf("creating shared memory\n");
        init_flag = 1;
    } else {
        printf("opening shared memory\n");
        fd = shm_open("fim", O_RDWR, S_IRWXU);
    }
    if (-1 == fd) {
        printf("fd is negative\n");
        abort();
    }
    if ((1 == init_flag) && -1 == ftruncate(fd, sizeof(fim_t))){
        printf("ftruncate failed %d\n", errno);
        abort();
    }

    fim_t *fim = mmap(NULL, sizeof(fim_t), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if(MAP_FAILED == fim){
        printf("mmap failed\n");
        abort();
    }
    if(init_flag){
        fim->num_procs = 1;
        my_rank = 0;
        for(int x=0;x<MAX_FIM_PROC;x++){
            fim->procs[x] = 0;
        }
        fim->numa_nodes[0] = &(fim->procs[0]);
    } else {
        my_rank = __sync_fetch_and_add(&(fim->num_procs),1);
        fim->procs[my_rank] = my_rank;
        fim->numa_nodes[0] = &(fim->procs[my_rank]);
    }
    printf("my rank is: %"PRId64"\n",my_rank);
    sleep(5);
    printf("my numa val is %"PRId64"\n",*fim->numa_nodes[0]);
    printf("rank %"PRId64" is going down\n", my_rank);
    // SHUTDOWN
    uint64_t active = __sync_sub_and_fetch(&(fim->num_procs),1);
    printf("num active is now %"PRId64"\n", active);
    close(fd);
    shm_unlink("fim");
    return 0;
}

What I expect/hope to happen would be that I run one process then immediately start another and the first process prints "my numa val is 1" (due to the second process setting the numa_node[0] value) and both exit cleanly. However, the second process runs fine, but in the first process seg faults (memory access) at the print statement for numa_node[0] (after the sleep).

So here's my question: Am I doing something wrong or is my approach unworkable? If it is unworkable, is there another way to achieve the result I'm looking for?

Upvotes: 3

Views: 1688

Answers (2)

Ziffusion
Ziffusion

Reputation: 8913

My understanding is that setting the numa_nodes references to addresses of the procs array should not be a memory access violation because their addresses are both entirely within the shared memory segment.

The problem is that different processes map the shared memory to different addresses.

fim_t *fim = mmap(NULL, sizeof(fim_t), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

fim will have different values in different processes. Print it out to check this.

This causes the pointers to the int64_t procs[MAX_FIM_PROC] elements to be different in different processes.

fim is <addr1> in process 1
fim is <addr2> in process 2

&fim->procs[0] will be different in two processes

&fim->procs[0] is <addr1> + <offset> in process 1
&fim->procs[0] is <addr2> + <offset> in process 2

Because these are different values, they cannot be shared between processes. Valid pointer in one process will not be valid in another process.

There are two possible solutions to this.

  1. Force the shared memory to map to the same address in all processes. mmap has an option to accomplish this. Then you can share pointers to elements in the shared memory across processes.
  2. Do not share pointers in shared memory. Share indexes instead.

Upvotes: 2

Arlie Stephens
Arlie Stephens

Reputation: 1176

You haven't done anything to arrange for all users of the shared memory to map it at the same virtual address. Some *nix systems will do this by default, but most will not.

Either try to map your segment at a fixed address (and deal with failure - this may not succeed) - or store offsets in the shared memory, not actual pointers.

Upvotes: 4

Related Questions