Compile external module for Android

Question

I am trying to compile a driver as a loadable module but whenever I adb shell into my phone and do an insmod test.ko I get the error message insmod: failed to load /data/local/tmp/test.ko: Exec format error. Grep'ing dmesg I find the following log: test: no symbol version for module_layout.

I've done quite a bit of googling [Ref] [Ref] [Ref] [Ref] [Ref] and reading the linux kbuild documentation txt files [Ref] [Ref] to no avail, so if someone knows the answer it'd be fab :) (I've only referenced the most helpful links I found).

What I've done so far is this:

I have an Qualcomm Aurora checkout and in the kernel directory I type the following

cp arch/arm64/configs/gemini_user_defconfig .config [B]# Has CONFIG_MODVERSIONS enabled and MODULE_SIG*=n[/B]
yes "" | make oldconfig ARCH=arm64 CROSS_COMPILE=aarch64-linux-android- V=1
make prepare ARCH=arm64 CROSS_COMPILE=aarch64-linux-android- V=1
make scripts ARCH=arm64 CROSS_COMPILE=aarch64-linux-android- V=1
make modules ARCH=arm64 CROSS_COMPILE=aarch64-linux-android- V=1

I have my PATH set to point to /MyAOSBDir/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin so I pick up the correct toolchain.

At this point appears to be well. In my kernel directory I have Module.symvers, so I appear to be set.

Now to the directory I have set up outside of this build tree with a dummy test... The Makefile looks like this:

KERNEL_DIR:=/solomon-build/MiNote2AOSB/kernel/
obj-m += test.o
PWD := $(shell pwd)

.PHONY: all
all:
        $(MAKE) M=$(PWD) ARCH=arm64 CROSS_COMPILE=aarch64-linux-android- -C $(KERNEL_DIR) modules V=1
clean:
        $(MAKE) M=$(PWD) -C $(KERNEL_DIR) clean

The driver is just a dummy:

#include <linux/module.h>       /* Needed by all modules */
#include <linux/kernel.h>       /* Needed for KERN_INFO */
#include <linux/init.h>         /* Needed for the macros */
static int __init hello_start(void)
{
  printk(KERN_INFO "Hello world\n");
  return 0;
}
static void __exit hello_end(void)
{
  printk(KERN_INFO "Goodbye world\n");
}
module_init(hello_start);
module_exit(hello_end);

I do make make all at the comment line and get this output:

make M=/solomon-build/build_mxt_kmod ARCH=arm64 CROSS_COMPILE=aarch64-linux-android- -C /solomon-build/MiNote2AOSB/kernel/ modules V=1
make[1]: Entering directory `/solomon-build/MiNote2AOSB/kernel'
test -e include/generated/autoconf.h -a -e include/config/auto.conf || (                \
        echo >&2;                                                       \
        echo >&2 "  ERROR: Kernel configuration is invalid.";           \
        echo >&2 "         include/generated/autoconf.h or include/config/auto.conf are missing.";\
        echo >&2 "         Run 'make oldconfig && make prepare' on kernel src to fix it.";      \
        echo >&2 ;                                                      \
        /bin/false)
mkdir -p /solomon-build/build_mxt_kmod/.tmp_versions ; rm -f /solomon-build/build_mxt_kmod/.tmp_versions/*
make -f ./scripts/Makefile.build obj=/solomon-build/build_mxt_kmod
  ./scripts/gcc-wrapper.py aarch64-linux-android-gcc -Wp,-MD,/solomon-build/build_mxt_kmod/.test.o.d  -nostdinc -isystem /solomon-build/MiNote2AOSB/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/../lib/gcc/aarch64-linux-android/4.9.x-google/include -I./arch/arm64/include -Iarch/arm64/include/generated  -Iinclude -I./arch/arm64/include/uapi -Iarch/arm64/include/generated/uapi -I./include/uapi -Iinclude/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -mlittle-endian -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -mgeneral-regs-only -fno-pic -fno-delete-null-pointer-checks -Os -Wno-maybe-uninitialized --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector -Wno-unused-but-set-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-var-tracking-assignments -g -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time  -DMODULE  -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(test)"  -D"KBUILD_MODNAME=KBUILD_STR(test)" -c -o /solomon-build/build_mxt_kmod/.tmp_test.o /solomon-build/build_mxt_kmod/test.c
(cat /dev/null;   echo kernel//solomon-build/build_mxt_kmod/test.ko;) > /solomon-build/build_mxt_kmod/modules.order
make -f ./scripts/Makefile.modpost
  find /solomon-build/build_mxt_kmod/.tmp_versions -name '*.mod' | xargs -r grep -h '\.ko$' | sort -u | sed 's/\.ko$/.o/' | scripts/mod/modpost -m  -i ./Module.symvers -I /solomon-build/build_mxt_kmod/Module.symvers  -o /solomon-build/build_mxt_kmod/Module.symvers -S -E -w  -s -T -
  ./scripts/gcc-wrapper.py aarch64-linux-android-gcc -Wp,-MD,/solomon-build/build_mxt_kmod/.test.mod.o.d  -nostdinc -isystem /solomon-build/MiNote2AOSB/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/../lib/gcc/aarch64-linux-android/4.9.x-google/include -I./arch/arm64/include -Iarch/arm64/include/generated  -Iinclude -I./arch/arm64/include/uapi -Iarch/arm64/include/generated/uapi -I./include/uapi -Iinclude/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -mlittle-endian -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -mgeneral-regs-only -fno-pic -fno-delete-null-pointer-checks -Os -Wno-maybe-uninitialized --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector -Wno-unused-but-set-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-var-tracking-assignments -g -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time  -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(test.mod)"  -D"KBUILD_MODNAME=KBUILD_STR(test)" -DMODULE  -c -o /solomon-build/build_mxt_kmod/test.mod.o /solomon-build/build_mxt_kmod/test.mod.c
  aarch64-linux-android-ld -EL -r  -T ./scripts/module-common.lds --build-id   --fix-cortex-a53-843419 -o /solomon-build/build_mxt_kmod/test.ko /solomon-build/build_mxt_kmod/test.o /solomon-build/build_mxt_kmod/test.mod.o
make[1]: Leaving directory `/solomon-build/MiNote2AOSB/kernel'

So, there are no messages saying there is a missing Module.symvers, so that's good, and I do get a built .ko file.

What I don't understand is the message ERROR: Kernel configuration is invalid. The files it is talking about exist, so I know they have been created in my kernel build.

Anyway, trying to insmod this onto the phone gives the message I mentioned previously. If I cat Module.symvers I see:

0xff924338      backlight_force_update  drivers/video/backlight/backlight       EXPORT_SYMBOL
0x30254daf      test_iosched_register   block/test-iosched      EXPORT_SYMBOL
<snip>
0xdd502540      backlight_device_unregister     drivers/video/backlight/backlight       EXPORT_SYMBOL
0x9939eba0      backlight_unregister_notifier   drivers/video/backlight/backlight       EXPORT_SYMBOL

So no symbol module_layout in the file, and thus no CRC, which is what I presume is creating the error message?

Any clues would be much appreciated, thank you.

Oh I should also mention that the module and kernel vermagic match:

# modinfo /data/local/tmp/test.ko
filename:       /data/local/tmp/test.ko
depends:
vermagic:       3.18.20-mytestkernel-perf-g671c431-dirty SMP preempt mod_unload modversions aarch64

# uname -a
Linux localhost 3.18.20-mytestkernel-perf-g671c431-dirty #1 SMP PREEMPT Mon Jun 5 15:46:13 BST 2017 aarch64

*EDIT - FOUND SOLUTION - NEW QUESTON *: I've gotten much further since posting. What I realised was, was that the compilation of the kernel using the Android build system is producing a radically different Module.symvers one from my method above (for some reason it took a while for the penny to drop and for me to check the Andoird build output - duh!). In fact, I've realised I always had the Module.symvers file, it was just located in ../out/target/product/msm8996/obj/KERNEL_OBJ/Module.symvers relative to the kernel dir. Copying this file into kernel and re-running my module build now produces a module that loads correctly :)

So, I guess the question becomes, why do the two files differ? What is Android doing differently? I assume it is setting up some more detailed config, but how/what/why?

Answer 1

So, the answer was relatively simple in the end...

I just use the normal Android build process of sourcing envsetup.sh, running lunch and then building the kernel:

. build/envsetup.sh
lunch msm8996-userdebug
make kernel -j16

(Remember to make sure in your .config file to set CONFIG_MODULES=y and set any kernel drivers to =m if you want them as modules).

In the separate directory, not in the Android build tree I now have my new module and the following Makefile, which now looks like this:

MY_ANDROID_ROOT_DIR :=/path/to/my/android/checkout
KERNEL_DIR:=$(MY_ANDROID_ROOT_DIR)/kernel
obj-m += my_driver.o
PWD := $(shell pwd)

.PHONY: all
all:
        $(MAKE) \
            M=$(PWD) \
            ARCH=arm64 \
            CROSS_COMPILE=aarch64-linux-android- \
            -C $(KERNEL_DIR) \
            modules \
            V=1 \
            O=$(MY_ANDROID_ROOT_DIR)/out/target/product/msm8996/obj/KERNEL_OBJ
clean:
        $(MAKE) M=$(PWD) -C $(KERNEL_DIR) clean

The make all output will still be in your modules directory (not the Android output dir).

This is basically identical to all of the examples relating to pure Linux builds. The only extra bit of information I needed to provide was where the Android build system output its build files, which is O=$(MY_ANDROID_ROOT_DIR)/out/target/product/<your-product>/obj/KERNEL_OBJ.

The O= define tells Make where to find the output of the kernel build.

The 'V=1` define tells Make to output verbose build information.

modules is a kernel target that will build you module defined in the Makefile variable obj-m. This variable "specifies the object files which are built as loadable kernel modules".

Now my modules build fine and I don't need to worry about independently compiling the kernel or anything like that to fix missing file messages (although it will still output about autoconf.h, which I don't understand because it exists in my tree and doesn't seem to stop my modules working).

PS If you module includes more than one file define obj-m += my_driver.o and then my_driver-m:= <extra objects>.