Using different library versions: fixing inconsistent assumptions over interface

Question

The issue

After creating a library my_lib.cm*a which depends on sqlite3.cm*a, trying to build a new project using my_lib.cm*a but an older version of sqlite3.cm*a I get a compile time error "inconsistent assumptions over interface Sqlite3". A similar error occurs when trying to use my_lib.cma with the older sqlite3.cma in the toplevel. The two different versions are actually on different systems, with my_lib.cm*a being copied to the older one.

Testing and exploration of issue

I ran a diff of the two sqlite3.mli files and identified that the issue seems to be one line. In the newer version of sqlite3 it is:

external enable_load_extension :
  db -> bool -> bool = "caml_sqlite3_enable_load_extension"

but has the "noalloc" option on the older system:

  external enable_load_extension :
  db -> bool -> bool = "caml_sqlite3_enable_load_extension" "noalloc"

So what I did was copy the newer version of sqlite3.mli to the system with the older sqlite3 (in a scratch directory), compiled it into sqlite3.cmi, copied the old sqlite3.cma and sqlite3.cmxa into the scratch directory. Now, if in the toplevel I do

#load "sqratch/dir/sqlite3.cma"
#load "my_lib.cma"
my_lib.do_stuff

it suddenly works - no errors get reported. I can also compile a program prog.ml that uses my_lib.cma with ocamlc sratch/dir/sqlite.cma my_lib.cma prog.ml -o prog and it compiles without error and runs just fine.

Though I don't really understand exactly how the compiler makes use of the interface files with the byte code files, from this it seems to me that a byte code library uses a .cmi file to define to interface and doesn't include any interface information itself, so the behaviour I've described so far seems to make sense.

Where I get confused is when I try to use the native compiler. If I try ocamlopt sratch/dir/sqlite.cmxa my_lib.cmxa prog.ml -o prog then the compiler again complains about my_lib.cmxa and sratch/dir/sqlite3.cmxa making inconsistent assumptions over the interface Sqlite3. From this I deduce that native compilation units (is that the right term?) or at least native archives contain the interface information in them. This would seem weird to me though, as the manual doesn't say anything about cmxa files having interfaces included in any way (though it does talk about other files types being included).

So now my for my questions...

• Are my deductions correct?
• Is my hack for the toplevel/byte code compiler (i.e. editing the mli to the expected one and then using that) something that will generally/often work or have I stumbled across a rare case where it does.
• Is there a similar hack to get native compilation to work?
• Any good suggestions for references about all this sort of compiler business? (I have struggled to find good references for how all the compilation/linking works) Everything I find just doesn't seem to really explain stuff (even the manual which I would have expected to be some enormous reference document that would be completely incomprehensible to me.) Maybe there aren't really references of the type I'm looking for and I'd have to learn how the C compiler works for/instead? (I've only written "hello world" level programs in C before and ocaml the first language I've used to produce a native executable).
• Is there a standard way to make libraries more system independent (that does not depend on opam), like somehow including sqlite3.cmxa in my_lib.cmxa? (I would think using -for-pack/-pack but I need the actual sqlite3.ml files for this don't I?)
• Is this behaviour in some way specific to the external function (I really don't know anything about interfacing ocaml with C)?.
• This is me being super lazy at this point (because I don't have the energy to go looking through documentation for sqlite3 that isn't the sqlite3.mli and its not particularly relevant), but if someone knows off the top of their head; what does the "noalloc" actually do? Without knowing actually what the external keyword is/does I'm assuming that "noalloc" is an argument for an external function in a C library, but I don't know what the pros/cons of using it might be.

As a final note, I know that this won't be the 'right' way to handle this situation; I suppose the usual thing to do would be to use opam to switch to the same compiler that was used to make my_lib.cm*a and then use opam to install the same version of sqlite3, but that isn't what I'm looking for (mostly because I'm looking to understand the compile process a little better, but also opam doesn't seem to work/it spits out errors when I tried to install it on the older system). Basically, I'd say I'm not looking for answers that boil down to "use opam on the older system".

Edits

• Well a few minutes more work yielded a decent solution (which I guess is basically the 'normal'/obvious way to do this) to the problem, which I suppose should generally work (unless the part of the interface of the external library actually changes). Building my_lib.cm*a from source on the older system makes everything work. I guess this should my newness to compiling/distributing software, though this doesn't answer some of the "conceptual" questions.

Answer 1

Are my deductions correct?

More or less.

Is my hack for the toplevel/byte code compiler (i.e. editing the mli to the expected one and then using that) something that will generally/often work or have I stumbled across a rare case where it does.

You stumbled a rare case. Actually you just provided more information, that allows a compiler to call this external function more efficiently. In general it shouldn't work of course, as you break the consistency between the implementation and its interface.

Is there a similar hack to get native compilation to work?

Well you can recompile the cmxa (library) file the same way you recompiled cmi file. But it will be already not a hack.

Any good suggestions for references about all this sort of compiler business?

The compiler code itself. There is an OCaml Compiler Hacking wiki, that contains some amount of useful information, but they don't cover linking.

Is there a standard way to make libraries more system independent (that does not depend on opam), like somehow including sqlite3.cmxa in my_lib.cmxa?

There is no standard way, but you can just copy all the files into a folder. (Btw, cmxa doesn't contain the binary code, it is in .a file. The cmxa as well as cmx just contains eXtra information about a compilation unit or units).

I would think using -for-pack/-pack...

for-pack and pack are designed to solve namespace problems, underneath the hood, the pack is still the same set of cmxa, cmx, a and o files.

but I need the actual sqlite3.ml files for this don't I?

Technically yes, unless you're going to use compiler tools to hack it.

Is this behaviour in some way specific to the external function (I really don't know anything about interfacing ocaml with C)?.

No. The inconsistency check just compares md5 sums of compiled interfaces and implementations.

... but if someone knows off the top of their head; what does the "noalloc" actually do?

noalloc designates to the compiler that this external C functions doesn't allocate any OCaml values. That means that a compiler doesn't need to insert a special prologue and epilogue codes for GC frame table, when it calls the function. This actually makes a call very fast, just an assembly call instruction. This qualifier should be documented in the next release of OCaml (4.03).

From this I deduce that native compilation units (is that the right term?) or at least native archives contain the interface information in them.

Yes it is a right term. And yes, they contain some information about the interface: the name and md5sum of imported interfaces. You can use ocamlobjinfo program to dump this information.

Though I don't really understand exactly how the compiler makes use of the interface files with the byte code files, from this it seems to me that a byte code library uses a .cmi file to define to interface and doesn't include any interface information itself, so the behaviour I've described so far seems to make sense.

Library code contains at least md5 sums of interfaces. You just bypassed the consistency check, that is made in the linking stage, and break compiler's assumptions, that if a unit is checked vs some cmi, then no one will just substitute this cmi later. So the cma files still thinks that it uses an old cmi.