PXE boot fail with kernel panic: Unable to mount root fs

Question

I am having an issue on some servers since some time, and fail to find the issue. These are x86_64 server, with Intel Xeon, configured to boot in UEFI over network, through an iPXE rom. Kernel and initramfs are the ones from Centos 8 (tried 8.0 and 8.2).

But when booting, I always end up with (on every servers, so should not be related to an hardware failure):

[    5.542304] hid-generic 0003:0557:2221.0002: input,hidraw1: USB HID v1.00 Keyboard [Winbond Electronics Corp Hermon USB hidmouse Device] on usb-0000:00:1a.0-1.3/input1
[    5.599611] rtc_cmos 00:02: setting system clock to 2020-06-26 19:21:43 UTC (1593199303)
[    5.620965] md: Waiting for all devices to be available before autodetect
[    5.640580] md: If you don't use raid, use raid=noautodetect
[    5.659949] md: Autodetecting RAID arrays.
[    5.676869] md: autorun ...
[    5.691838] md: ... autorun DONE.
[    5.707883] List of all partitions:
[    5.723667] No filesystem could mount root, tried:
[    5.723667]
[    5.754724] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    5.775237] CPU: 9 PID: 1 Comm: swapper/0 Not tainted 4.18.0-193.6.3.el8_2.x86_64 #1
[    5.815778] Call Trace:
[    5.830360]  dump_stack+0x5c/0x80
[    5.845920]  panic+0xe7/0x2a9
[    5.860995]  mount_block_root+0x2c5/0x2e9
[    5.877407]  ? do_early_param+0x91/0x91
[    5.892617]  prepare_namespace+0x135/0x16b
[    5.907676]  kernel_init_freeable+0x22e/0x258
[    5.922607]  ? rest_init+0xaa/0xaa
[    5.937398]  kernel_init+0xa/0xff
[    5.950991]  ret_from_fork+0x35/0x40
[    5.964572] Kernel Offset: 0x2ca00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    6.000387] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

The iPXE boot script is:

#!ipxe

kernel http://10.10.0.1/vmlinuz-4.18.0-193.6.3.el8_2.x86_64 initrd=initramfs-4.18.0-193.6.3.el8_2.x86_64.img selinux=0  rd.shell rd.debug root=live:http://10.10.0.1/squashfs.img rw console=tty0 console=ttyS1,115200
initrd http://10.10.0.1/initramfs-4.18.0-193.6.3.el8_2.x86_64.img

boot

Which generate this in the console:

http://10.10.0.1/vmlinuz-4.18.0-193.6.3.el8_2.x86_64... ok
http://10.10.0.1/initramfs-4.18.0-193.6.3.el8_2.x86_64.img... ok
INTEL 0x6f080f70 MAC reset (081c0261/80280783 was 081c0261/80280783)
INTEL 0x6f080f70 MAC reset (081c0261/80280783 was 081c0261/80280783)
INTEL 0x6f081ab0 MAC reset (081c0261/80280787 was 081c0261/80280787)
[    0.000000] Linux version 4.18.0-193.6.3.el8_2.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP Wed Jun 10 11:09:32 UTC 2020
[    0.000000] Command line: vmlinuz-4.18.0-193.6.3.el8_2.x86_64 initrd=initramfs-4.18.0-193.6.3.el8_2.x86_64.img selinux=0 rd.shell rd.debug root=live:http://10.10.0.1/squashfs.img rw console=tty0 console=ttyS1,115200
...

On the server side, on apache logs I have:

10.10.2.1 - - [26/Jun/2020:19:55:42 +0200] "GET /vmlinuz-4.18.0-193.6.3.el8_2.x86_64 HTTP/1.1" 200 8913656 "-" "iPXE/1.0.0+"
10.10.2.1 - - [26/Jun/2020:19:55:42 +0200] "GET /initramfs-4.18.0-193.6.3.el8_2.x86_64.img HTTP/1.1" 200 53703611 "-" "iPXE/1.0.0+"

So it seems to be working perfectly.

This is diskless boot here, but whatever I try (kickstart diskfull install, or even kernel+initrd alone) I always end up to this kernel panic...

I tried to reset BIOS settings, try to boot in legacy/pcbios instead of UEFI, tried to desactivate sata disks, etc. Always this same error. Also tried to use kernel+initrd from Centos ISO (checked checksum), tried to use the ones from my management node. Nothing. Do I miss something obvious?

Does any of you have an idea or already faced this kind of issue?

Many thanks in advance :-)

With my best regards

Beuk