Design pattern for managing shared libraries loaded into memory

Question

gcc (GCC) 4.7.2

Hello,

I am developing a large project that will contain 2 modules (shared libraries) that I will develop.

The modules are shared libraries created by me in C, that will have to sync and exchange messages between each other.

The manager module will control and load both these modules (.so) into memory. If one was to fail. The manager could try and re-load it.

I am wondering as this is my first time I have done something like this. Is there any design patterns that could be followed?

All this will be written in C and using the APR (Apache Portable Runtime) for memory pool management and maybe some threading pool if needed.

1. Start manager that will load both modules.
2. Manager then calls some functions on both of them maybe to start and stop them, and possible cleanup.
3. Once both modules have been loaded and started. They should be able to exchange some messages between each other.

The modules will all run on the same machine running Redhat.

Many thanks for any suggestions.

Answer 1

One critical question: why do you want to implement this in this fashion? You are 'tightly' coupling what are essentially 'loosely' coupled components (because shared libraries have all sorts of crash-related issues: they'll take the manager down). Why not have a Manager program that (can) launch and relaunch if necessary 2 or more child processes.

Have the child processes communicate with the Manager, or with each other, using some sort of protocol. I would recommend ZeroMQ both because it is awesome and because it totally hides the interprocess communication, so it could be sockets (between different machines), or inter-process between threads, or named pipes on a single machine, which is very fast. This means that after implementing your clients, you can decide how you want to deploy them: as shared libraries loaded into the manager, as separate processes running on the same box, or as distributed processes running on separate machines, and you'll hardly need to change anything. Which means very scalable.

However, if you are dedicated to the shared library approach, there is one 'design pattern' that I would absolutely recommend for this, although it can be a bit tricky to implement. But it'll be worth its weight.

Your manager, before passing messages between modules, should check their timestamps and, if anything has changed, recompile and reload them. This will mean that your code changes are 'hot': you won't have to stop the manager, recompile, and restart the manager to see the changes. So you can program in C more like one develops in js! It will save you hours and hours.

I did something similar a while ago (not the inter-library comms) using C++ and APR. The code is a bit 'hacky', but here it is anyway ;-)

Note that it depends on having a Makefile for each submodule in its own directory, and because of dependencies, I don't check timestamps, I just recompile on each request. This might not be ideal for you, so you might need to rethink that part.

The hardest part of getting it to work was getting the right permissions on directories, but come to think of it, that was because I was running this as an fcgi process, so when it actually ran it did so as the webserver. You will most likely not encounter those issues.

#ifndef _CMJ_RUN_HPP
#define _CMJ_RUN_HPP

#include <fcgio.h>
#include <stdlib.h>

#include <iostream>
#include <string>
#include <sstream>
#include <vector>

#include <apr.h>
#include <apr_dso.h>
#include <apr_pools.h>
#include <apr_thread_proc.h>

#include <boost/filesystem.hpp>
#include <boost/algorithm/string.hpp>
#include <boost/algorithm/string/case_conv.hpp>

#include <cgicc/Cgicc.h>
#include <cgicc/HTTPHTMLHeader.h>
#include <cgicc/HTMLClasses.h>
#include <stdexcept>

#include <cstdarg>

class Line {
protected:
    std::stringstream line_;
    bool isError_;
public:
    Line(const char* line, bool isError) : line_(line), isError_(isError) {}
    Line(const Line& rhs) : line_(rhs.line()), isError_(rhs.error()) {}
    bool error() const { return isError_; }
    const char* line() const { return line_.str().c_str(); }
    const Line& operator = (const Line& rhs) {
        line_.str() = rhs.line();
        isError_ = rhs.error();
        return rhs;
    }
};

class Run {
protected:
    int exitCode_;
    std::vector<Line> out_;
    bool errors_;
protected:
    void run(const char* dir, const char* cmd, std::vector<const char*> &args, apr_pool_t* parentPool) ;
public:
    Run(const char* dir, const char* cmd, std::vector<const char*> &args, apr_pool_t* parentPool);
    Run(const char* dir, const char* cmd, apr_pool_t* parentPool);
    int exitCode() { return exitCode_; }
    bool errors() { return errors_; }
    bool errors(std::ostream& out);
    int size() { return out_.size(); }
    Line& line(int i) { return out_[i]; }
};

class dso_error: public std::runtime_error {
public:
    dso_error(const char* c) : std::runtime_error(c) {};
    dso_error(std::string err) : std::runtime_error(err) {};
    static dso_error instance(const char* format, ...) {
        char errbuf[8192];
        va_list va;
        va_start(va, format);
        vsnprintf(errbuf, 8192, format, va);
        va_end(va);
        return dso_error(errbuf);
    }
};

/**
 * Provides a building and loading framework for Dynamic libraries, with the full power
 * of make behind it.
 * Usage:
 * <code>
 * DsoLib so("/var/www/frontier/echo","/var/www/frontier/echo/libecho.so",pool);
 * if (!so.errors(outStream)) {
 *  void (*pFn)(void) = sym("initialize");
 *  (*pFn)();
 * }
 * </code>
 */
class DsoLib : public Run {
protected:
    apr_pool_t* pool_;
    apr_dso_handle_t* dso_;
    std::string dirname_;
    std::string libname_;
public:
    /** dir is the directory where make should be executed, libname is full path to the library
     * from current working directory.
     */
    DsoLib(const char* dir, const char* libname, apr_pool_t* parentPool) throw(dso_error);
    ~DsoLib();
    void* sym(const char* symbol) throw (dso_error);
    void* sym(std::string symbol) throw (dso_error) { return sym(symbol.c_str()); }
};

#endif

And Run.cpp

#include "Run.hpp"

#include <string>
#include <sstream>
#include <boost/filesystem.hpp>
#include <cassert>

#define DBGENDL " (" << __FILE__ << ":" << __LINE__ << ")" << endl


using namespace std;

Run::Run(const char* dir, const char* cmd, apr_pool_t* pool) : errors_(false) {
    vector<const char *> args;
    run(dir, cmd, args, pool);
}

Run::Run(const char* dir, const char* cmd, vector<const char*> &args, apr_pool_t* pool) : errors_(false) {
    run(dir, cmd, args, pool);
}

void
Run::run(const char* dir, const char* cmd, vector<const char*> &args, apr_pool_t* parentPool) {
    cout << "Run::run(dir=" << ", cmd=" << cmd << ", args...)" << endl;
    apr_status_t status;
    char aprError[1024];
    struct aprPool_s {
        apr_pool_t* pool_;
        aprPool_s(apr_pool_t* parent) {
            apr_pool_create(&pool_, parent);
        }
        ~aprPool_s() {
            apr_pool_destroy(pool_);
        }
        operator apr_pool_t*  () { return pool_; }
    } pool (parentPool);

    apr_procattr_t* attr;
    if (APR_SUCCESS != (status = apr_procattr_create(&attr, pool))) {
        cerr << "apr_procattr_create error: " << apr_strerror(status, aprError, sizeof(aprError)) << endl;
    }
    if (APR_SUCCESS != (status = apr_procattr_dir_set(attr, dir))) {
        cerr << "apr_procattr_dir_set error: " << apr_strerror(status, aprError, sizeof(aprError)) << endl;
    }
    if (APR_SUCCESS != (status = apr_procattr_cmdtype_set(attr, APR_PROGRAM_ENV))) {
        cerr << "apr_procattr_cmdtype_set error: " << apr_strerror(status, aprError, sizeof(aprError)) << endl;
    }
    if (APR_SUCCESS != (status = apr_procattr_io_set(attr, APR_NO_PIPE, APR_FULL_NONBLOCK, APR_FULL_NONBLOCK))) {
        cerr << "apr_procattr_io_set error: " << apr_strerror(status, aprError, sizeof(aprError)) << endl;
    }
    if (APR_SUCCESS != (status = apr_procattr_user_set(attr, "craig", "lateral"))) {
        cerr << "apr_procattr_user_set error: " << apr_strerror(status, aprError, sizeof(aprError)) << endl;
    }
    if (APR_SUCCESS != (status = apr_procattr_group_set(attr, "craig"))) {
        cerr << "apr_procattr_group_set error: " << apr_strerror(status, aprError, sizeof(aprError)) << endl;
    }
    apr_proc_t proc;

    const char **argv = (const char**) new char*[ 2 + args.size() ];
    argv[0] = cmd;
    size_t i=0;
    size_t argc=args.size();
    for (i=0; i<argc; i++) {
        argv[i+1] = args[i];
        cerr << "arg " << i << " = " << args[i];
    }
    argv[i+1] = NULL;
    argc++;
    cerr << "About to execute " << cmd << " in dir " << dir << endl;
    cerr << "ARGS:" << endl;
    for (i=0; i<argc; i++) {
        cerr << "[" << i << "]: " << argv[i] << endl;
    }

    if (APR_SUCCESS != (status = apr_proc_create(&proc, cmd, argv, NULL, attr, pool))) {
        cerr << "apr_proc_create error: " << apr_strerror(status, aprError, sizeof(aprError)) << endl;
    }

    apr_exit_why_e exitWhy;
    cerr << "--- " << cmd << " ---" << endl;
    while (APR_CHILD_NOTDONE == (status = apr_proc_wait(&proc, &exitCode_, &exitWhy, APR_NOWAIT))) {
        char line[1024];
        status = apr_file_gets(line, sizeof(line), proc.out);
        if (APR_SUCCESS==status) {
            out_.push_back(Line(line, false));
            cerr << line << endl;
        }

        status = apr_file_gets(line, sizeof(line), proc.err);
        if (APR_SUCCESS==status) {
            out_.push_back(Line(line, true));
            errors_ = true;
            cerr << "E:" << line ;
        }
    }
    cerr << " -----" << endl;

    delete[] argv;

    if ( (APR_CHILD_DONE != status) && (APR_PROC_EXIT != status) ) {
        cerr << "apr_proc_wait error: " << apr_strerror(status, aprError, sizeof(aprError)) << endl;
    }
    cerr << cmd << " exited " << ((APR_PROC_EXIT==exitWhy) ? "PROC_EXIT" :
            ((APR_PROC_SIGNAL==exitWhy) ? "PROC_SIGNAL" :
            ((APR_PROC_SIGNAL_CORE==exitWhy) ? "PROC_SIGNAL_CORE" : "Unknown"))) << endl;
}

bool
Run::errors(std::ostream& os) {
    cerr << "Run::errors(ostream) : errors()=" << errors() << endl;
    if (errors()) {
        cerr << "Writing errors to ostream" << endl;
        os << "Content-type: text/html\r\n\r\n";
        os << "<html><head><title>Errors</title>"
            << "<link rel=\"stylesheet\" type=\"text/css\" href=\"css/frontier.css\"></link>"
            << "</head>"
            << "<body>";
        for (int i=0; i<size(); i++) {
            Line& errline = line(i);
            os << "<div class=\"" << ( (errline.error() ? "error" : "out" ) ) << "\">"
                    << errline.line()
                    << "</div>";
        }
        os
            << "</body>"
            << "</html>";
    }
    return errors();
}

DsoLib::DsoLib(const char* dir, const char* libname, apr_pool_t* parentPool) throw (dso_error) :
    Run(dir, "/usr/bin/make", parentPool), pool_(NULL), dso_(NULL), dirname_(dir), libname_(libname)
{
    if (errors()) {
        cerr << "Run encountered errors, quitting DsoLib::DsoLib()" << DBGENDL;
        //throw dso_error::instance("Build failed for dir %s, library %s", dir, libname);
        return;
    } else {
        cerr << "No errors encountered with Run in DsoLib::DsoLib" << DBGENDL;
    }

    apr_status_t status;
    if (APR_SUCCESS != apr_pool_create(&pool_, parentPool)) {
        cerr << "Failed to allocate pool" << DBGENDL;
        throw dso_error("Failed to allocate apr_pool");
    }

    cerr << "Created pool ok" << DBGENDL;   //(" << __FILE__ << ":" << __LINE__ << ")" << endl;

    if (APR_SUCCESS != (status = apr_dso_load(&dso_, libname, pool_))) {
        cerr << "apr_dso_load(" << libname << ") failed" << DBGENDL;
        char aprError[1024];
        throw dso_error::instance("dso_load failed, path=%s, error=%s",
                libname, apr_strerror(status, aprError, sizeof(aprError)));
    }
    cerr << "Loaded dso ok" << DBGENDL;
#if 0
    void (*initialize)(apr_pool_t*) = reinterpret_cast< void(*)(apr_pool_t*) > (sym("initialize"));
    if (initialize) {
        cerr << "found initialize sym: about to call initialize" << DBGENDL;
        initialize(pool_);
        cerr << "initialize(pool) returned ok" << DBGENDL;
    } else {
        cerr << "initialize sym not found" << DBGENDL;
    }
#endif
    cerr << "Exiting DsoLib::DsoLib(" << dir << ", " << libname << ") with success." << endl;
}

DsoLib::~DsoLib() {
    cerr << "Entering DsoLib::~DsoLib(dir=" << dirname_ <<", " << "lib=" << libname_ << ")" << endl;
    if (NULL!=dso_) {
        void (*terminate)(void) = reinterpret_cast<void(*)()>(sym("terminate"));
        if (terminate) terminate();
        apr_status_t status = apr_dso_unload(dso_);
        if (APR_SUCCESS != status) {
            char err[8192];
            cerr << "ERR apr_dso_unload failed: " << apr_dso_error(dso_, err, sizeof(err)) << endl;
        } else {
            cerr << "Unloaded " << libname_ << endl;
        }
    } else {
        cerr << "ERR dso_ handle is NULL" << endl;
    }
    if (NULL!=pool_) apr_pool_destroy(pool_);
}

void *
DsoLib::sym(const char* symbol) throw (dso_error) {
    cerr << "sym('" << symbol << "')" << DBGENDL;
    cerr << "dso_ == NULL ? " << ((NULL==dso_)?"true":"false") << DBGENDL;
    cerr << "dso_ = " << dso_ << DBGENDL;
    assert(NULL!=symbol);
    assert(NULL!=dso_);
    apr_status_t status;
    void* p = NULL;
    if (APR_SUCCESS != (status = apr_dso_sym((apr_dso_handle_sym_t*)&p, dso_, symbol))) {
        cerr << "apr_dso_sym() DID NOT RETURN APR_SUCCESS" << DBGENDL;
        char aprError[1024];
        stringstream err;
        err << "dso_sym failed, symbol=" << symbol << ": " << apr_strerror(status, aprError, sizeof(aprError));
        cerr << err.str() << DBGENDL;
    } else {
        cerr << "sym succeeded for " << symbol << " in " << libname_ << DBGENDL;
    }
    return p;
}

Answer 2

The architecture here is relatively simple so you do not need a complex design pattern.

The main problem is data integrity. If the system crashes partially, how do you ensure that both have the same copy of the data?

Since you are using messaging you have half solved the problem already. You only need to do two things:

(1) store the list of recent messages and create a rollback/update mechanism to restore a module given a checkpoint backup and the list of messages since the checkpoint

(2) make sure the messages are atomic; ie you never want a partial message or transaction to be accepted because if the sender crashes in the middle of sending a message the receiver could be corrupted by accepting incomplete information.

To solve problem 2 add a checksum or hash to the end of a transaction. The receiver does not finalize its acceptance of a message set unless the hash is received and matches the data.

Answer 3

If you have already decided to use APR, you should probably use the dynamic library loading it provides. You can find a tutorial here.

Answer 4

Here after a simple example of project based in your request:

the architecture of your source code could be like this:

src
   |__handler1.c //containing the main function
   |__handler2.c //containing other functions
   |__lib1.c //containing lib1 source
   |__lib2_file1.c  //containing lib2 source
   |__lib2_file2.c  //containing lib2 source
   |__Makefile  // file which contains commands to build the project
   |__inc
         |__lib1.h
         |__lib2.h
         |__handler2.h

handler1.c

#include <stdio.h>
#include "lib1.h"
#include "lib2.h"
#include "handler2.h"

int main()
{
    char *s1, *s2;
    print_hello_from_handler2();
    s1 = get_message_from_lib1_method1();
    get_message_from_lib1_method2(&s2);

    printf("s1 = %s\n",s1);
    printf("s2 = %s\n",s2);
    printf("extern string_from_lib1 = %s\n",string_from_lib1);
    printf("extern string_from_lib2 = %s\n",string_from_lib2);
}

handler2.c

#include <stdio.h>

void print_hello_from_handler2()
{
    printf("hello world from handler2\n");
}

lib1.c

#include "lib2.h"
char *string_from_lib1="message from lib1 variable";

char *get_message_from_lib1_method1()
{
    return get_message_from_lib2_method1();
}

void get_message_from_lib1_method2(char **s)
{
    get_message_from_lib2_method2(s);
}

lib2_file1.c

char *string_from_lib2="message from lib2 variable";

char *str="message from lib2 method1";

char *get_message_from_lib2_method1()
{
    return str;
}

lib2_file2.c

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

void get_message_from_lib2_method2(char **s)
{
    *s = malloc(30);
    strcpy(*s,"message from lib2 method2");
}

lib1.h

extern char *string_from_lib1;

char *get_message_from_lib1_method1();
void get_message_from_lib1_method2(char **s);

lib2.h

extern char *string_from_lib2;

char *get_message_from_lib2_method1();
void get_message_from_lib2_method2(char **s);

handler2.h

void print_hello_from_handler2();

Makefile

SHLIB_EXT=so
LINK=$(CC)
SHLIB1_FILE=libmodule1.$(SHLIB_EXT).1
SHLIB2_FILE=libmodule2.$(SHLIB_EXT).1
SHLIB1_FLAGS=-shared -Wl,-soname,$(SHLIB1_FILE)
SHLIB2_FLAGS=-shared -Wl,-soname,$(SHLIB2_FILE)
FPIC=-fPIC

all: libmodule2.$(SHLIB_EXT) libmodule1.$(SHLIB_EXT) handler


%.o: %.c
    $(CC) -Iinc -c -o $@ $^

handler: handler1.o handler2.o
    $(CC) -o $@ $^ -L. -lmodule2 -lmodule1

lib2_file1.o: lib2_file1.c
    $(CC) $(FPIC) -Iinc -c -o $@ $<

lib2_file2.o: lib2_file2.c
    $(CC) $(FPIC) -Iinc -c -o $@ $<

libmodule2.$(SHLIB_EXT): lib2_file1.o lib2_file2.o
    $(LINK) $(SHLIB2_FLAGS) -o $(SHLIB2_FILE) $^
    ln -sf $(SHLIB2_FILE) $@

libmodule1.o: lib1.c
    $(CC) $(FPIC) -Iinc -c -o $@ $<

libmodule1.$(SHLIB_EXT): libmodule1.o
    $(LINK) $(SHLIB1_FLAGS) -o $(SHLIB1_FILE) $< -L. -lmodule2
    ln -sf $(SHLIB1_FILE) $@


clean:
    rm -f *.o *.so* handler
    rm -f /usr/lib/$(SHLIB1_FILE)
    rm -f /usr/lib/$(SHLIB2_FILE)
    rm -f /usr/lib/libmodule1.$(SHLIB_EXT)
    rm -f /usr/lib/libmodule2.$(SHLIB_EXT)

install:
    cp $(SHLIB1_FILE) /usr/lib/
    cp $(SHLIB2_FILE) /usr/lib/
    cp handler /usr/bin/
    ln -sf /usr/lib/$(SHLIB1_FILE) /usr/lib/libmodule1.$(SHLIB_EXT)
    ln -sf /usr/lib/$(SHLIB2_FILE) /usr/lib/libmodule2.$(SHLIB_EXT)

the command to compile your project

linux$ cd src
linux$ make

and then install the binary and the libraries

linux$ sudo make install

to clean installed libraries and the binary and to clean build binary libraries and objects:

linux$ sudo make clean

To run the application:

linux$ handler
hello world from handler2
s1 = message from lib2 method1
s2 = message from lib2 method2
extern string_from_lib1 = message from lib1 variable
extern string_from_lib2 = message from lib2 variable
linux$

Answer 5

As I get it, you need to decouple point 1 and 2.

• For that you should have a separate class called BootstrapManager, which will be responsible to loading the modules and reloading if they fail.
• Next you need is an abstract class called Module which would have 3 methods,
  start() - starts a module, stop() - stops a module, cleanUp() - cleanup activities, communicate() - communicates with another module.
• Now both Module1 and Module 2 will extend this class and implement their own business logic accordingly.

Answer 6

The manager module will control and load both these modules (.so) into memory. If one was to fail. The manager could try and re-load it.

This is usually a bad idea if it's in a single C process - if one of these modules fail, you're unlikely to be able to safely unload it, much less load it again. If you need to be able to recover from module failure, you must use independent processes. The code can still be in an .so file though - just fork() the manager once for each module to load; this is the model used by the chrome plugins API, for example.

Moreover, dealing with component failure can be very, very tricky in itself. Just because A restarts doesn't mean B is ready to talk to a newly restarted A. You may want to try to glean some ideas off erlang, which handles component failure exceptionally well by encouraging the decomposition of applications into message-passing subcomponents with a hierarchy of supervisor modules to restart failing components. This may be a bit overkill if you only have two modules, but it's something to think about at least.

As for how to communicate, there are a number of paradigms. If these modules are in the same process, you could just pass a vtable around. That is, for example:

// moduleA.h

struct vtable_A {
  void (*do_something)();
};

void set_vtable_B(struct vtable_B *);
struct vtable_A *get_vtable_A();
void start_A();

// moduleB.h
struct vtable_B {
  void (*do_something)();
};

void set_vtable_A(struct vtable_A *);
struct vtable_B *get_vtable_B();
void start_B();

Your manager would load both, pass the vtable from A to B and vice versa, and thereafter call the start routines. Be careful with ordering - either A must be started before B is ready, or vice versa, and they need to be okay with this.

If they're in independent processes, message passing is usually the way to go. It's essentially a network protocol at that point - your subprocesses will send serialized messages to the manager, and the manager routes them to other subprocesses. The conversation might look a bit like this:

MGR->A      START
MGR->B      START
A->MGR      REGISTER_ENDPOINT 'ProcessA'
A->MGR      WATCH_ENDPOINT 'ProcessB'
MGR->A      OK_REGISTER 'ProcessA'
MGR->A      OK_WATCH 'ProcessB'
B->MGR      REGISTER_ENDPOINT 'ProcessB'
B->MGR      WATCH_ENDPOINT 'ProcessA'
MGR->B      OK_REGISTER 'ProcessB'
MGR->A      NEW_ENDPOINT 'ProcessB'
A->MGR      APPLICATION_DATA TO:'ProcessB', PAYLOAD:"Hello, world!"
MGR->B      OK_WATCH 'ProcessA'
MGR->B      NEW_ENDPOINT 'ProcessA'
MGR->B      APPLICATION_DATA FROM:'ProcessA', PAYLOAD:"Hello, world!"

Keep in mind, there are many other ways to structure this kind of protocol than the example above, and to build RPC on top of a message-passing protocol. You may be interested in looking at things such as DBUS (which you may be able to use directly!) or DCOM, which have done this sort of thing before. Other optimizations on top of this sort of protocol include using the manager to establish a direct channel of some sort between A and B, and getting it involved again only if A or B need to be restarted.

That said, don't get too deep into the details of how the manager works before you figure out what it needs to do. Design the plugin<->manager high level interface, and plugin<->plugin protocol; only then design the details of the plugin<->manager interface. It's far too easy to get sidetracked and end up with something way too complex like CORBA or SOAP.

Answer 7

I'm a bit allergic to "pattern talk" but this is how I would approach this:

• Decide on threading model.

  • Will your modules exchange information using memory they control, or the manager?
  • Should the modules wait on condition variables shared between them or owned by the manager?

• Decide on how generic a manager you need.

  • Should it be able to poll a directory for modules or read a configuration or both?
  • If the manager manages messages, what is needed for signalling between modules?

When you know this, the rest should be mostly business logic that will live in the modules.