How to debug linux kernel module?

Question

I am following this link(Understanding a Kernel Oops!) to debug a kernel module

But the contents of the .init.text section has all zeroes. And therefore I am not able to see the exact line causing this crash.

My /var/log/syslog output is:

Aug  1 18:07:33 manik kernel: [  149.855117] BUG: unable to handle kernel paging request at 0000000000ef6850
Aug  1 18:07:33 manik kernel: [  149.855136] IP: [<ffffffffc07577b0>] register_scard+0x240/0x660 [scard]
Aug  1 18:07:33 manik kernel: [  149.855143] PGD 0
Aug  1 18:07:33 manik kernel: [  149.855146] Oops: 0000 [#1] SMP
Aug  1 18:07:33 manik kernel: [  149.855151] Modules linked in: scard(POE+) kmikohn(OE) fmon(OE) khcd(OE) kcid(OE) kspi(POE) kpwrfail(OE) snd_usb_audio input_leds joydev hid_multitouch snd_usbmidi_lib i2c_ocores intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mk7i_pci(OE) snd_hda_codec_hdmi irqbypass crct10dif_pclmul crc32_pclmul cryptd snd_soc_rt5640 snd_soc_rl6231 snd_soc_core lpc_ich snd_hda_intel snd_hda_codec snd_compress ac97_bus snd_pcm_dmaengine snd_hda_core ie31200_edac snd_hwdep edac_core shpchp snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq 8250_fintek snd_seq_device snd_timer dw_dmac dw_dmac_core elan_i2c snd soc_button_array snd_soc_sst_acpi spi_pxa2xx_platform soundcore i2c_designware_platform i2c_designware_core 8250_dw mac_hid parport_pc ppdev lp parport autofs4 nouveau mxm_wmi i915 wmi i2c_algo_bit ttm ahci drm_kms_helper e1000e syscopyarea sysfillrect sysimgblt ptp fb_sys_fops libahci pps_core drm sdhci_acpi i2c_hid sdhci video fjes hid_generic usbhid hid
Aug  1 18:07:33 manik kernel: [  149.855281] CPU: 0 PID: 3749 Comm: modprobe Tainted: P        W IOE   4.4.0-31-generic #50
Aug  1 18:07:33 manik kernel: [  149.855286] Hardware name: ADLINK Technology Inc. Express-HL./SHARKBAY, BIOS 1.14 01/01/2013
Aug  1 18:07:33 manik kernel: [  149.855290] task: ffff880233c844c0 ti: ffff8800b354c000 task.ti: ffff8800b354c000
Aug  1 18:07:33 manik kernel: [  149.855294] RIP: 0010:[<ffffffffc07577b0>]  [<ffffffffc07577b0>] register_scard+0x240/0x660 [scard]
Aug  1 18:07:33 manik kernel: [  149.855302] RSP: 0018:ffff8800b354fc30  EFLAGS: 00010006
Aug  1 18:07:33 manik kernel: [  149.855306] RAX: 0000000000ef6850 RBX: ffff8800b8b59400 RCX: 0000000000000006
Aug  1 18:07:33 manik kernel: [  149.855310] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff880240a0dc70
Aug  1 18:07:33 manik kernel: [  149.855314] RBP: ffff8800b354fc60 R08: 000000000000000a R09: 0000000000000786
Aug  1 18:07:33 manik kernel: [  149.855318] R10: ffffea0008cde400 R11: 0000000000000786 R12: 0000000000000000
Aug  1 18:07:33 manik kernel: [  149.855322] R13: ffff8800b8b59400 R14: 0000000000000000 R15: 0000000000000000
Aug  1 18:07:33 manik kernel: [  149.855326] FS:  00007f94b559a700(0000) GS:ffff880240a00000(0000) knlGS:0000000000000000
Aug  1 18:07:33 manik kernel: [  149.855333] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug  1 18:07:33 manik kernel: [  149.855337] CR2: 0000000000ef6850 CR3: 00000000b34cd000 CR4: 00000000001406f0
Aug  1 18:07:33 manik kernel: [  149.855341] Stack:
Aug  1 18:07:33 manik kernel: [  149.855343]  ffff8800b8b59480 ffffffff81e11080 ffff880231952a20 0000000000000000
Aug  1 18:07:33 manik kernel: [  149.855351]  ffffffffc0048000 ffff8800b3614a80 ffff8800b354fc88 ffffffffc0048045
Aug  1 18:07:33 manik kernel: [  149.855358]  0000000000000000 0000f42400000010 0000000095f455c1 ffff8800b354fd08
Aug  1 18:07:33 manik kernel: [  149.855366] Call Trace:
Aug  1 18:07:33 manik kernel: [  149.855370]  [<ffffffffc0048000>] ? 0xffffffffc0048000
Aug  1 18:07:33 manik kernel: [  149.855375]  [<ffffffffc0048045>] scard_module_init+0x45/0x1000 [scard]
Aug  1 18:07:33 manik kernel: [  149.855381]  [<ffffffff81002123>] do_one_initcall+0xb3/0x200
Aug  1 18:07:33 manik kernel: [  149.855388]  [<ffffffff811cf630>] ? __vunmap+0xc0/0xe0
Aug  1 18:07:33 manik kernel: [  149.855393]  [<ffffffff811eb813>] ? kmem_cache_alloc_trace+0x183/0x1f0
Aug  1 18:07:33 manik kernel: [  149.855398]  [<ffffffff811ec60a>] ? kfree+0x13a/0x150
Aug  1 18:07:33 manik kernel: [  149.855404]  [<ffffffff8118c763>] do_init_module+0x5f/0x1cf
Aug  1 18:07:33 manik kernel: [  149.855410]  [<ffffffff8110a1c7>] load_module+0x1667/0x1c00
Aug  1 18:07:33 manik kernel: [  149.855415]  [<ffffffff81106770>] ? __symbol_put+0x60/0x60
Aug  1 18:07:33 manik kernel: [  149.855420]  [<ffffffff81213150>] ? kernel_read+0x50/0x80
Aug  1 18:07:33 manik kernel: [  149.855425]  [<ffffffff8110a9a4>] SYSC_finit_module+0xb4/0xe0
Aug  1 18:07:33 manik kernel: [  149.855430]  [<ffffffff8110a9ee>] SyS_finit_module+0xe/0x10
Aug  1 18:07:33 manik kernel: [  149.855435]  [<ffffffff8182dd72>] entry_SYSCALL_64_fastpath+0x16/0x71
Aug  1 18:07:33 manik kernel: [  149.855439] Code: 48 c7 c7 a0 87 75 c0 e8 26 4b a3 c0 48 c7 c7 90 83 75 c0 4c 8b 7b 10 e8 16 4b a3 c0 48 8b 05 d8 7d e8 ff 05 50 08 00 00 4c 01 f8 <44> 8b 28 48 c7 c7 d8 83 75 c0 41 83 e5 08 e8 f4 4a a3 c0 c7 83
Aug  1 18:07:33 manik kernel: [  149.855510] RIP  [<ffffffffc07577b0>] register_scard+0x240/0x660 [scard]
Aug  1 18:07:33 manik kernel: [  149.855516]  RSP <ffff8800b354fc30>
Aug  1 18:07:33 manik kernel: [  149.855518] CR2: 0000000000ef6850
Aug  1 18:07:33 manik kernel: [  149.861630] ---[ end trace 80da0b0ded5071b5 ]---

Am I doing something wrong ?

Using printk I was able to narrow it down to the source line that was causing this. The source code causing this is :

uint8_t old_cmd = ioread32( base + SC_REG_ICC_COMMAND );

This line works fine with 32-bit linux kernel. The above crash is happening with 64-bit linux kernel 4.4.0 series ( running Ubuntu 16.0.4 LTS )

Answer 1

Another useful tool analyzing kernel is crash. It can be used to analyze kernel coredump or living system. Here is a good reading to start:

http://www.dedoimedo.com/computers/crash-analyze.html

Answer 2

I assume the module you are trying to run is called 'scard'?

If so, you can try to use objdump to see where in the code the following occurred: [ 552.027690] [] scard_module_init+0x45/0x1000 [scard]

you should find the C file containing this function (lets say it is scard.c), then run objudump on its o file: objdump -S [path to scard.o] | less

Then search for the scard_module_init function, see the address in hex and add to it 0x45 - skip to this line in the objdump and you will see the last assembly line which was run which caused the exception.

From what we can see in the backtrace, it should be some print which probably tries to access invalid memory.