CODEWITHSUNDEEP
linux-kerneldriverx86-64dma
dtoch
dtoch

Reputation: 11

dma_alloc_coherent failed on x86_64 but works on i686

I have a driver for pci device which uses CMA allocation mechanism for DMA allocations. It works fine on kernel 3.18 in 32bit mode but when I try to use it in 64 kernel(same config as in 32bit, but switched on 64bit mode) dma allocation failed.

Only thing I see in dmesg is:

 fallback device: swiotlb buffer is full (sz: 8388608 bytes)

I use kernel cmdline:

swiotlb=16384 iommu=soft cma=256M

and allocating 8Mb.

The function call is:

new_region->kaddr = dma_alloc_coherent( NULL, size, &new_region->paddr, GFP_KERNEL | GFP_DMA32 );

Can someone explain this behaviour in 64bit mode?

Upvotes: 1

Views: 3486

Answers (1)

Fengbo Wu
Fengbo Wu

Reputation: 41

After more investigation, I think you may have the same rootcause as me, just list it out for your reference. The CMA allocation will start from the highest memory blocks, so if you have more that 3G memory, the last physical memory will be above 0xFFFFFFFF, that means the CMA's base address is above 4GB, but dma_allocat_coherent() requires the address is below the mask [(0x1 << 32)-1] = 0xFFFFFFFF, if the alloacted dma end address is bigger than 0xFFFFFFFF, it will fallback to the swiotlb buffer and then you will see the error you described. Please see the below memory map, the last 2G is above 4GB memory space.

To resolve this problem, we can specify the cma start address, size and limit size to manually control the CMA reserved location.

e820: BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x00000000779c4fff] usable
BIOS-e820: [mem 0x00000000779c5000-0x0000000077a45fff] reserved
BIOS-e820: [mem 0x0000000077a46000-0x0000000079426fff] usable
BIOS-e820: [mem 0x0000000079427000-0x000000007b32efff] reserved
BIOS-e820: [mem 0x000000007b32f000-0x000000007b985fff] ACPI NVS
BIOS-e820: [mem 0x000000007b986000-0x000000007bad3fff] ACPI data
BIOS-e820: [mem 0x000000007bad4000-0x000000007bafffff] usable
BIOS-e820: [mem 0x000000007bb00000-0x000000008fffffff] reserved
BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
BIOS-e820: [mem 0x0000000100000000-0x000000017fffffff] usable

cma=nn[MG]@[start[MG][-end[MG]]]
            [ARM,X86,KNL]
            Sets the size of kernel global memory area for
            contiguous memory allocations and optionally the
            placement constraint by the physical address range of
            memory allocations. A value of 0 disables CMA
            altogether. For more information, see
            include/linux/dma-contiguous.h

To give you a workaround first, please add "mem=3072M" to your kernel command line. Then it will work in your case. The reason why you see this may be related to the NULL passed to the function dma_alloc_coherent(). It will use the x86_dma_fallback_dev by default. Will update more once I get more information.

Upvotes: 4

Related Questions