CODEWITHSUNDEEP
memorylinux-kernelmmu
Pea
Pea

Reputation: 1

Read and Write issue from PCIe memory once in Kernel

My setup

Currently, I am trying to access a PCIe Low memory region from the Xilinx MPSoC (which is mapped to a Egress BAR under the FSBL).The memory address is 0xE000_0000. This works fine, as long I stay in EL-3 or EL-2.

MPSoC PCIe in Endpoint mode. Since there is no EP support from the NWL drivers (only for RC), I decided to configure the PCIe bridge and and bars mapping in the FSBL. Also because we need to have the EP performing some operations before entering the kernel.

The issue

Once the kernel boots (in EL-1) I can not read or write from the egress memory mapped BAR that runs fine in U-boot or Baremetal(EL-2). I get immediately a "Kernel panic - not syncing: Asynchronous SError Interrupt". I tried both from a custom kernel device driver and also directly with devmen.

This is the error log output:

devmem 0xe8000000 32 0x000000A3
[   42.351077] SError Interrupt on CPU3, code 0x00000000bf000002 -- SError
[   42.351089] CPU: 3 PID: 681 Comm: devmem Tainted: G           O       6.6.40-xilinx-g2b7f6f70a62a #1
[   42.351096] Hardware name: xlnx,zynqmp (DT)
[   42.351098] pstate: 60000000 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   42.351105] pc : 00000000004b8990
[   42.351107] lr : 00000000004b7684
[   42.351108] sp : 0000ffffe64ba8e0
[   42.351110] x29: 0000ffffe64ba9a0 x28: 00000000004002e0 x27: 00000000005bfb08
[   42.351120] x26: 00000000005c8000 x25: 0000000000000018 x24: 0000000000000002
[   42.351127] x23: 0000000000000000 x22: 0000ffffe64bab70 x21: 0000ffffe64bab48
[   42.351135] x20: 0000000000000021 x19: 0000ffffe64bab48 x18: 0000000000000000
[   42.351142] x17: 0000000000000000 x16: 0000000000000000 x15: 000000000360ed96
[   42.351149] x14: 0000000000577cc5 x13: 0000000000000000 x12: 000000000d25ede0
[   42.351156] x11: 0000ffffe64ba9b0 x10: ffff00001a0acc7a x9 : 0000000000000010
[   42.351164] x8 : 000000000055a950 x7 : 000000000000000f x6 : 0000ffffe64bae1e
[   42.351170] x5 : 0000000000000003 x4 : 000000000055b250 x3 : 0000000000000000
[   42.351177] x2 : 0000000000000010 x1 : 00000000005c1000 x0 : 0000000000000000
[   42.351186] Kernel panic - not syncing: Asynchronous SError Interrupt
[   42.351189] CPU: 3 PID: 681 Comm: devmem Tainted: G           O       6.6.40-xilinx-g2b7f6f70a62a #1
[   42.351194] Hardware name: xlnx,zynqmp (DT)
[   42.351197] Call trace:
[   42.351199]  dump_backtrace+0x90/0xe8
[   42.351214]  show_stack+0x18/0x24
[   42.351222]  dump_stack_lvl+0x48/0x60
[   42.351230]  dump_stack+0x18/0x24
[   42.351236]  panic+0x314/0x370
[   42.351243]  nmi_panic+0x8c/0x90
[   42.351248]  arm64_serror_panic+0x6c/0x78
[   42.351254]  do_serror+0x28/0x68
[   42.351259]  __el0_error_handler_common+0x40/0xa4
[   42.351267]  el0t_64_error_handler+0x10/0x1c
[   42.351274]  el0t_64_error+0x190/0x194
[   42.351281] SMP: stopping secondary CPUs
[   42.351288] Kernel Offset: disabled
[   42.351290] CPU features: 0x0,00000008,00020000,0000420b
[   42.351293] Memory Limit: none
[   42.537059] ---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---

However, I can read pcie configuration space from the kernel in the Low PCIe memory from 0xE0000000 to 0xE0000FFD.

Afterwards I get Bus error (core dumped)

root@root:~# devmem 0xe0000000 
0xD01110EE

The error cause, I'm assuming, could be one of two causes:

  1. missing or incorrect memory region mapping in the kernel. Perhaps a MMU mapping issue ?
  2. Accessing an uninitialized or unsupported address region in the kernel. Perhaps marked as secure memory from the ATF?

What have I tried so far

  1. U-boot: add an extra mmapping at ext-local-src/uboot_src/arch/arm/mach-zynqmp/cpu.c.

https://github.com/Xilinx/u-boot-xlnx/blob/38eb17fa2bd1db2f2abc6ea1132a61962011d9cf/arch/arm/mach-zynqmp/cpu.c#L46

static struct mm_region zynqmp_mem_map[ZYNQMP_MEM_MAP_MAX] = {
    {
        .virt = 0x80000000UL,
        .phys = 0x80000000UL,
        .size = 0x70000000UL,
        .attrs = PTE_BLOCK_MEMTYPE(MT_DEVICE_NGNRNE) |
             PTE_BLOCK_NON_SHARE |
             PTE_BLOCK_PXN | PTE_BLOCK_UXN
    }, {
        .virt = 0xf8000000UL,
        .phys = 0xf8000000UL,
        .size = 0x07e00000UL,
        .attrs = PTE_BLOCK_MEMTYPE(MT_DEVICE_NGNRNE) |
             PTE_BLOCK_NON_SHARE |
             PTE_BLOCK_PXN | PTE_BLOCK_UXN
    }, {
        .virt = 0x400000000UL,
        .phys = 0x400000000UL,
        .size = 0x400000000UL,
        .attrs = PTE_BLOCK_MEMTYPE(MT_DEVICE_NGNRNE) |
             PTE_BLOCK_NON_SHARE |
             PTE_BLOCK_PXN | PTE_BLOCK_UXN
    }, {
        .virt = 0x1000000000UL,
        .phys = 0x1000000000UL,
        .size = 0xf000000000UL,
        .attrs = PTE_BLOCK_MEMTYPE(MT_DEVICE_NGNRNE) |
             PTE_BLOCK_NON_SHARE |
             PTE_BLOCK_PXN | PTE_BLOCK_UXN
    }, {
        .virt = 0xe0000000UL,
        .phys = 0xe0000000UL,
        .size = 0x10000000UL,
        .attrs = PTE_BLOCK_MEMTYPE(MT_DEVICE_NGNRNE) | PTE_BLOCK_NON_SHARE | PTE_BLOCK_PXN | PTE_BLOCK_UXN
    }

and increase the array the size to 5

https://github.com/Xilinx/u-boot-xlnx/blob/38eb17fa2bd1db2f2abc6ea1132a61962011d9cf/arch/arm/mach-zynqmp/cpu.c#L46

#define ZYNQMP_MEM_MAP_USED 5

Output log:

Setting MMU...
MMU Table:
Table count ranges :15
V: 0x0080000000 P: 0x0080000000 Size: 0x0070000000 Attr: 0x0060000000000000 
V: 0x00f8000000 P: 0x00f8000000 Size: 0x0007e00000 Attr: 0x0060000000000000 
V: 0x0400000000 P: 0x0400000000 Size: 0x0400000000 Attr: 0x0060000000000000 
V: 0x1000000000 P: 0x1000000000 Size: 0xf000000000 Attr: 0x0060000000000000 
V: 0x00e0000000 P: 0x00e0000000 Size: 0x0010000000 Attr: 0x0060000000000000 
V: 0x00ffe00000 P: 0x00ffe00000 Size: 0x0000200000 Attr: 0x0000000000000310 
V: 0x0000000000 P: 0x0000000000 Size: 0x007ff00000 Attr: 0x0000000000000310 
V: 0x0800000000 P: 0x0800000000 Size: 0x0180000000 Attr: 0x0000000000000310
  1. ATF: add an extra entry in the plat_zynqmp_mmap     atf_src/plat/xilinx/zynqmp/aarch64/zynqmp_common.c
const mmap_region_t plat_zynqmp_mmap[] = {
    { PCIE_LOW_BASE, PCIE_LOW_BASE, PCIE_LOW_SIZE, MT_MEMORY | MT_RW | MT_NS },
    { DEVICE0_BASE, DEVICE0_BASE, DEVICE0_SIZE, MT_DEVICE | MT_RW | MT_SECURE },
    { DEVICE1_BASE, DEVICE1_BASE, DEVICE1_SIZE, MT_DEVICE | MT_RW | MT_SECURE },
    { CRF_APB_BASE, CRF_APB_BASE, CRF_APB_SIZE, MT_DEVICE | MT_RW | MT_SECURE },
    {0}
};

None of the two above tries worked and maybe I am over complicating. I would appreciate very much any help or guidance.

Upvotes: 0

Views: 44

Answers (0)

Related Questions