Reading and writing in-memory data

Question

In the following example, I am taking a struct that occupies 32 bytes in memory and writing that to a file and reading it back in -- i.e., serializing the data to a binary format:

#include <stdio.h>

typedef struct _Person {
    char   name[20];
    int    age;
    double weight;
} Person;

int main(void)
{

    Person tom = (Person) {.name="Tom", .age=20, .weight=125.0};

    // write the struct to a binary file
    FILE *fout = fopen("person.b", "wb");
    fwrite(&tom, sizeof tom, 1, fout);
    fclose(fout);

    // read the binary data and set the person to that
    Person unknown;
    FILE *fin = fopen("person.b", "rb");
    fread(&unknown, sizeof unknown, 1, fin);
    fclose(fin);

    // confirm all looks ok
    printf("{name=%s, age=%d, weight=%f}", unknown.name, unknown.age, unknown.weight);

}

Note however that these are all values on the stack, and no pointers/indirection is involved. How might data be serialized to a file when, for example, multiple pointers can be involved, multiple variables may point to the same memory location, etc. Is this effectively what protocol buffers do?

Answer 1

Ok so you want a binary file. I used to do it this way long ago. It's fine. It just breaks when you move to another platform or bitness. I'm teaching the old way because it's a good place to start. Newer ways are popular now because they keep working when changing platforms or bitnesses.

When writing records to the file I would use structs like this:

typedef struct _Person {
    char   name[20];
    int    age;
    double weight;
} Person;

typedef struct _Thing {
    char name[20];
};

typedef struct _Owner {
    int personId;
    int thingId;
} Owner;

See how the Owner structure has Id members. These are just indices into the arrays of the other structures.

These can be written out to a file one after another, usually prefixed by a single integer written directly that says how many records of each kind. The reader just allocates an array of structs with malloc big enough to hold them. As we add more items in memory we resize the arrays with realloc. We can (and should) also mark for deleting (say by setting the first character of name to 0) and reusing the record later.

The writer looks something like this:

void writeall(FILE *h, Person *allPeople, int nPeople, Thing *allThings, int nThings, Owner *allOwners, int nOwners)
{
    // Error checking omitted for brevity
    fwrite(&nPeople, sizeof(nPeople), 1, h);
    fwrite(allPeople, sizeof(*allPeople), nPeople, h);
    fwrite(&nThings, sizeof(nThings), 1, h);
    fwrite(allThings, sizeof(*allThings), nThings, h);
    fwrite(&nOwners, sizeof(nOwners), 1, h);
    fwrite(allOwners, sizeof(*allOwners), nOwners, h);
}

The reader in turn looks like this:

int writeall(FILE *h, Person **allPeople, int *nPeople, int *aPeople, Thing **allThings, int *nThings, int *aThings, Owner **allOwners, int *nOwners, int *aOwners)
{
    *aPeople = 0; // Don't crash on bad read
    *aThigns = 0;
    *aOwners = 0;
    *allPeople = NULL;
    *allThings = NULL;
    *allOwners = NULL;

    if (1 != fread(nPeople, sizeof(*nPeople), 1, h)) return 0;
    *allPeople = malloc(sizeof(**allPeople) * *nPeople);
    if (!allPeople) return 0; // OOM
    *aPeople = *nPeople;
    if (*nPeople != fread(*allPeople, sizeof(**allPeople), nPeople, h)) return 0;

    if (1 != fread(nThings, sizeof(*nThings), 1, h)) return 0;
    *allThings = malloc(sizeof(**allThings) * *nThings);
    if (!allThings) return 0; // OOM
    *aThings = *nThings;
    if (*nThings != fread(*allThings, sizeof(**allThings), nThings, h)) return 0;

    if (1 != fread(nOwners, sizeof(*nOwners), 1, h)) return 0;
    *allOwners = malloc(sizeof(**allOwners) * *nOwners);
    if (!allOwners) return 0; // OOM
    *aOwners = *nOwners;
    if (*nOwners != fread(*allOwners, sizeof(**allOwners), nOwners, h)) return 0;

    return 1;
}

There was an old technique for writing a heap arena directly to disk and reading it back again. I recommend never using it, and never storing pointers on disk. That way lies security nightmares.

When memory was cheap I would have talked about how to use block allocation and linked blocks to dynamically update the on-disk records partially; but now for problems you're going to encounter at this level I say don't bother, and just read the whole thing into RAM and write it back out again. Eventually you will learn databases, which handles that stuff for you.