Cannot reserve 512MB or more of CMA

Question

I'm trying to reserve 512MB of CMA memory on an ARM64 box with 64GB of memory, and I'll get "cma: Failed to reserve 512MiB" error message during linux boot. Reserving 384MB works fine. Please let me know if you need further info. I'd appreciate any help.

[ESL_Start_OS]:[644L] Start to jump Linux kernel
    Booting Linux on physical CPU 0x10000
    Initializing cgroup subsys cpuset
    Initializing cgroup subsys cpu
    Initializing cgroup subsys cpuacct
    Linux version 4.1.0+ (s00327669@salem-linux) (gcc version 4.9.3 20141031 (prerelease) (Linaro GCC 2014.11) ) #4 SMP Mon Oct 5 12:00:57 EDT 2015
    CPU: AArch64 Processor [411fd071] revision 1
    Detected PIPT I-cache on CPU0
    alternatives: enabling workaround for ARM erratum 832075
    earlycon: Early serial console at MMIO32 0x60300000 (options '')
    bootconsole [uart0] enabled
    efi: Getting EFI parameters from FDT:
    efi: UEFI not found.
    cma: Failed to reserve 512 MiB
    PERCPU: Embedded 16 pages/cpu @ffffffd7bfa00000 s27648 r8192 d29696 u65536
    Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 16514064
    Kernel command line: rdinit=/init console=ttyS0,115200 earlycon=uart8250,mmio32,0x60300000 cma=512M                                                                                                 
    log_buf_len individual max cpu contribution: 4096 bytes
    log_buf_len total cpu_extra contributions: 126976 bytes
    log_buf_len min size: 16384 bytes
    log_buf_len: 262144 bytes
    early log buf free: 14220(86%)
    PID hash table entries: 4096 (order: 3, 32768 bytes)
    Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
    Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
    software IO TLB [mem 0x3bbff000-0x3fbff000] (64MB) mapped at [ffffffc03bbff000-ffffffc03fbfefff]
    Memory: 65469716K/67104768K available (6109K kernel code, 537K rwdata, 2516K rodata, 436K init, 226K bss, 1635052K reserved, 0K cma-reserved)
    Virtual kernel memory layout:
        vmalloc : 0xffffff8000000000 - 0xffffffbdbfff0000   (   246 GB)
        vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000   (     8 GB maximum)
                  0xffffffbdc0000000 - 0xffffffbe20000000   (  1536 MB actual)
        fixed   : 0xffffffbffabfd000 - 0xffffffbffac00000   (    12 KB)
        PCI I/O : 0xffffffbffae00000 - 0xffffffbffbe00000   (    16 MB)
        modules : 0xffffffbffc000000 - 0xffffffc000000000   (    64 MB)
        memory  : 0xffffffc000000000 - 0xffffffd800000000   ( 98304 MB)
          .init : 0xffffffc0008ef000 - 0xffffffc00095c000   (   436 KB)
          .text : 0xffffffc000080000 - 0xffffffc0008ee7c4   (  8634 KB)
          .data : 0xffffffc000965000 - 0xffffffc0009eb400   (   537 KB)
    SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=32, Nodes=1
    Hierarchical RCU implementation.
            Additional per-CPU info printed with stalls.
    NR_IRQS:64 nr_irqs:64 0
    ITS: /interrupt-controller@6d000000/interrupt-controller@6c000000
    ITS: allocated 65536 Devices @17b9480000 (psz 4K, shr 2)
    ITS: allocated 512 Virtual CPUs @17b9418000 (psz 4K, shr 2)
    ITS: allocated 512 Interrupt Collections @17b9419000 (psz 4K, shr 2)
    ITS: /interrupt-controller@6d000000/interrupt-controller@4c000000
    ITS: allocated 65536 Devices @17b9500000 (psz 4K, shr 2)
    ITS: allocated 512 Virtual CPUs @17b9432000 (psz 4K, shr 2)
    ITS: allocated 512 Interrupt Collections @17b9433000 (psz 4K, shr 2)
    ITS: /interrupt-controller@6d000000/interrupt-controller@c6000000
    ITS: allocated 65536 Devices @17b9580000 (psz 4K, shr 2)
    ITS: allocated 512 Virtual CPUs @17b943d000 (psz 4K, shr 2)
    ITS: allocated 512 Interrupt Collections @17b943e000 (psz 4K, shr 2)
    ITS: /interrupt-controller@6d000000/interrupt-controller@a3000000
    ITS: allocated 65536 Devices @17b9600000 (psz 4K, shr 2)
    ITS: allocated 512 Virtual CPUs @17b9467000 (psz 4K, shr 2)
    ITS: allocated 512 Interrupt Collections @17b9468000 (psz 4K, shr 2)
    ITS: /interrupt-controller@6d000000/interrupt-controller@b7000000
    ITS: allocated 65536 Devices @17b9700000 (psz 4K, shr 2)
    ITS: allocated 512 Virtual CPUs @17b9681000 (psz 4K, shr 2)
    ITS: allocated 512 Interrupt Collections @17b9682000 (psz 4K, shr 2)
    GIC: using LPI property table @0x00000017b96a0000
    ITS: Allocated 1792 chunks for LPIs
    CPU0: found redistributor 10000 region 1:0x000000004d100000
    CPU0: using LPI pending table @0x00000017b96b0000
    MBIGEN: /peripherals/interrupt-controller@4c030000
    MBIGEN: /peripherals/interrupt-controller@6c030000
    MBIGEN: /peripherals/interrupt-controller@c6030000
    MBIGEN: /peripherals/interrupt-controller@a3030000
    MBIGEN: /peripherals/interrupt-controller@b7030000
    Architected cp15 timer(s) running at 50.00MHz (phys).
    clocksource arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
    sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
    Console: colour dummy device 80x25
    Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=200000)
    pid_max: default: 32768 minimum: 301
    Security Framework initialized
    Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes)
    Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes)
    Initializing cgroup subsys memory
    Initializing cgroup subsys devices
    Initializing cgroup subsys freezer
    Initializing cgroup subsys hugetlb
    hw perfevents: enabled with arm/armv8-pmuv3 PMU driver, 7 counters available
    EFI services will not be available.
    CPU1: Booted secondary processor
    Detected PIPT I-cache on CPU1
    CPU1: found redistributor 10001 region 1:0x000000004d130000
    CPU1: using LPI pending table @0x00000017b8ee0000
    CPU2: Booted secondary processor
    Detected PIPT I-cache on CPU2
    CPU2: found redistributor 10002 region 1:0x000000004d160000
    CPU2: using LPI pending table @0x00000017b8f10000
    CPU3: Booted secondary processor
    Detected PIPT I-cache on CPU3
    CPU3: found redistributor 10003 region 1:0x000000004d190000
    CPU3: using LPI pending table @0x00000017b8f40000
    CPU4: Booted secondary processor
    Detected PIPT I-cache on CPU4
    CPU4: found redistributor 10100 region 1:0x000000004d1c0000
    CPU4: using LPI pending table @0x00000017b8f80000
    CPU5: Booted secondary processor
    Detected PIPT I-cache on CPU5
    CPU5: found redistributor 10101 region 1:0x000000004d1f0000
    CPU5: using LPI pending table @0x00000017b8fa0000
    CPU6: Booted secondary processor
    Detected PIPT I-cache on CPU6
    CPU6: found redistributor 10102 region 1:0x000000004d220000
    CPU6: using LPI pending table @0x00000017b8fd0000
    CPU7: Booted secondary processor
    Detected PIPT I-cache on CPU7
    CPU7: found redistributor 10103 region 1:0x000000004d250000
    CPU7: using LPI pending table @0x00000017b8800000
    CPU8: Booted secondary processor
    Detected PIPT I-cache on CPU8
    CPU8: found redistributor 10200 region 1:0x000000004d280000
    CPU8: using LPI pending table @0x00000017b8840000
    CPU9: Booted secondary processor
    Detected PIPT I-cache on CPU9
    CPU9: found redistributor 10201 region 1:0x000000004d2b0000
    CPU9: using LPI pending table @0x00000017b8870000
    CPU10: Booted secondary processor
    Detected PIPT I-cache on CPU10
    CPU10: found redistributor 10202 region 1:0x000000004d2e0000
    CPU10: using LPI pending table @0x00000017b88b0000
    CPU11: Booted secondary processor
    Detected PIPT I-cache on CPU11
    CPU11: found redistributor 10203 region 1:0x000000004d310000
    CPU11: using LPI pending table @0x00000017b88e0000
    CPU12: Booted secondary processor
    Detected PIPT I-cache on CPU12
    CPU12: found redistributor 10300 region 1:0x000000004d340000
    CPU12: using LPI pending table @0x00000017b8910000
    CPU13: Booted secondary processor
    Detected PIPT I-cache on CPU13
    CPU13: found redistributor 10301 region 1:0x000000004d370000
    CPU13: using LPI pending table @0x00000017b8940000
    CPU14: Booted secondary processor
    Detected PIPT I-cache on CPU14
    CPU14: found redistributor 10302 region 1:0x000000004d3a0000
    CPU14: using LPI pending table @0x00000017b8970000
    CPU15: Booted secondary processor
    Detected PIPT I-cache on CPU15
    CPU15: found redistributor 10303 region 1:0x000000004d3d0000
    CPU15: using LPI pending table @0x00000017b89a0000
    CPU16: Booted secondary processor
    Detected PIPT I-cache on CPU16
    CPU16: found redistributor 20000 region 0:0x000000006d100000
    CPU16: using LPI pending table @0x00000017b89e0000
    CPU17: Booted secondary processor
    Detected PIPT I-cache on CPU17
    CPU17: found redistributor 20001 region 0:0x000000006d130000
    CPU17: using LPI pending table @0x00000017b8a10000
    CPU18: Booted secondary processor
    Detected PIPT I-cache on CPU18
    CPU18: found redistributor 20002 region 0:0x000000006d160000
    CPU18: using LPI pending table @0x00000017b8a40000
    CPU19: Booted secondary processor
    Detected PIPT I-cache on CPU19
    CPU19: found redistributor 20003 region 0:0x000000006d190000
    CPU19: using LPI pending table @0x00000017b8a80000
    CPU20: Booted secondary processor
    Detected PIPT I-cache on CPU20
    CPU20: found redistributor 20100 region 0:0x000000006d1c0000
    CPU20: using LPI pending table @0x00000017b8aa0000
    CPU21: Booted secondary processor
    Detected PIPT I-cache on CPU21
    CPU21: found redistributor 20101 region 0:0x000000006d1f0000
    CPU21: using LPI pending table @0x00000017b8ad0000
    CPU22: Booted secondary processor
    Detected PIPT I-cache on CPU22
    CPU22: found redistributor 20102 region 0:0x000000006d220000
    CPU22: using LPI pending table @0x00000017b8b10000
    CPU23: Booted secondary processor
    Detected PIPT I-cache on CPU23
    CPU23: found redistributor 20103 region 0:0x000000006d250000
    CPU23: using LPI pending table @0x00000017b8b40000
    CPU24: Booted secondary processor
    Detected PIPT I-cache on CPU24
    CPU24: found redistributor 20200 region 0:0x000000006d280000
    CPU24: using LPI pending table @0x00000017b8b70000
    CPU25: Booted secondary processor
    Detected PIPT I-cache on CPU25
    CPU25: found redistributor 20201 region 0:0x000000006d2b0000
    CPU25: using LPI pending table @0x00000017b8bb0000
    CPU26: Booted secondary processor
    Detected PIPT I-cache on CPU26
    CPU26: found redistributor 20202 region 0:0x000000006d2e0000
    CPU26: using LPI pending table @0x00000017b8be0000
    CPU27: Booted secondary processor
    Detected PIPT I-cache on CPU27
    CPU27: found redistributor 20203 region 0:0x000000006d310000
    CPU27: using LPI pending table @0x00000017b8400000
    CPU28: Booted secondary processor
    Detected PIPT I-cache on CPU28
    CPU28: found redistributor 20300 region 0:0x000000006d340000
    CPU28: using LPI pending table @0x00000017b8450000
    CPU29: Booted secondary processor
    Detected PIPT I-cache on CPU29
    CPU29: found redistributor 20301 region 0:0x000000006d370000
    CPU29: using LPI pending table @0x00000017b8470000
    CPU30: Booted secondary processor
    Detected PIPT I-cache on CPU30
    CPU30: found redistributor 20302 region 0:0x000000006d3a0000
    CPU30: using LPI pending table @0x00000017b84b0000
    CPU31: Booted secondary processor
    Detected PIPT I-cache on CPU31
    CPU31: found redistributor 20303 region 0:0x000000006d3d0000
    CPU31: using LPI pending table @0x00000017b84e0000
    Brought up 32 CPUs
    SMP: Total of 32 processors activated.
    CPU: All CPU(s) started at EL2
    alternatives: patching kernel code
    devtmpfs: initialized
    DMI not present or invalid.
    clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
    atomic64_test: passed
    pinctrl core: initialized pinctrl subsystem
    NET: Registered protocol family 16
    cpuidle: using governor ladder
    cpuidle: using governor menu
    vdso: 2 pages (1 code @ ffffffc00096d000, 1 data @ ffffffc00096c000)
    hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
    DMA: preallocated 256 KiB pool for atomic allocations
    Serial: AMBA PL011 UART driver
    hisi-smmu 40040000.smmu_pa: registered 0 master devices
    hisi-smmu 40040000.smmu_pa: probing hardware configuration...
    hisi-smmu 40040000.smmu_pa:     nested translation
    hisi-smmu 40040000.smmu_pa:     coherent table walk
    hisi-smmu 40040000.smmu_pa:     39-bit VA, 39-bit IPA, 48-bit PA
    hisi-smmu 60040000.smmu_pc: registered 0 master devices
    hisi-smmu 60040000.smmu_pc: probing hardware configuration...
    hisi-smmu 60040000.smmu_pc:     nested translation
    hisi-smmu 60040000.smmu_pc:     coherent table walk
    hisi-smmu 60040000.smmu_pc:     39-bit VA, 39-bit IPA, 48-bit PA
    hisi-smmu c0040000.smmu_dsa: registered 0 master devices
    hisi-smmu c0040000.smmu_dsa: probing hardware configuration...
    hisi-smmu c0040000.smmu_dsa:    nested translation
    hisi-smmu c0040000.smmu_dsa:    coherent table walk
    hisi-smmu c0040000.smmu_dsa:    39-bit VA, 39-bit IPA, 48-bit PA
    Warning: Mbigen device id already exist.Old:0x2,new:0x20
    hisi-smmu a0040000.smmu_m3: registered 0 master devices
    hisi-smmu a0040000.smmu_m3: probing hardware configuration...
    hisi-smmu a0040000.smmu_m3:     nested translation
    hisi-smmu a0040000.smmu_m3:     coherent table walk
    hisi-smmu a0040000.smmu_m3:     39-bit VA, 39-bit IPA, 48-bit PA
    hisi-smmu b0040000.smmu_pcie: registered 0 master devices
    hisi-smmu b0040000.smmu_pcie: probing hardware configuration...
    hisi-smmu b0040000.smmu_pcie:   nested translation
    hisi-smmu b0040000.smmu_pcie:   coherent table walk
    hisi-smmu b0040000.smmu_pcie:   39-bit VA, 39-bit IPA, 48-bit PA
    vgaarb: loaded
    SCSI subsystem initialized
    usbcore: registered new interface driver usbfs
    usbcore: registered new interface driver hub
    usbcore: registered new device driver usb
    Warning: Mbigen device id already exist.Old:0x1,new:0x10
    Warning: Mbigen device id already exist.Old:0x3,new:0x30
    Warning: Mbigen device id already exist.Old:0x3,new:0x30
    Switched to clocksource arch_sys_counter
    NET: Registered protocol family 2
    TCP established hash table entries: 524288 (order: 10, 4194304 bytes)
    TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
    TCP: Hash tables configured (established 524288 bind 65536)
    UDP hash table entries: 32768 (order: 8, 1048576 bytes)
    UDP-Lite hash table entries: 32768 (order: 8, 1048576 bytes)
    NET: Registered protocol family 1
    RPC: Registered named UNIX socket transport module.
    RPC: Registered udp transport module.
    RPC: Registered tcp transport module.
    RPC: Registered tcp NFSv4.1 backchannel transport module.
    Unpacking initramfs...
    Freeing initrd memory: 409600K (ffffffc007000000 - ffffffc020000000)
    futex hash table entries: 8192 (order: 7, 524288 bytes)
    audit: initializing netlink subsys (disabled)
    audit: type=2000 audit(5.048:1): initialized
    HugeTLB registered 2 MB page size, pre-allocated 0 pages
    VFS: Disk quotas dquot_6.6.0
    VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
    NFS: Registering the id_resolver key type
    Key type id_resolver registered
    Key type id_legacy registered
    fuse init (API version 7.23)
    9p: Installing v9fs 9p2000 file system support
    bounce: pool size: 64 pages
    Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
    io scheduler noop registered
    io scheduler cfq registered (default)
    hisi-pcie b0080000.pcie: Link up
    PCI host bridge /pcie@0xb0080000 ranges:
      MEM 0xb4100000..0xb4ffffff -> 0xb4100000
    hisi-pcie b0080000.pcie: PCI host bridge to bus 0000:10
    pci_bus 0000:10: root bus resource [bus 10-1f]
    pci_bus 0000:10: root bus resource [mem 0xb4100000-0xb4ffffff]
    pci 0000:10:00.0: IOMMU is currently not supported for PCI
    pci 0000:10:00.0: of_irq_parse_pci() failed with rc=-22
    pci 0000:10:00.0: Primary bus is hard wired to 0
    pci 0000:10:00.0: bridge configuration invalid ([bus 01-01]), reconfiguring
    pci 0000:11:00.0: IOMMU is currently not supported for PCI
    pci 0000:11:00.0: of_irq_parse_pci() failed with rc=-22
    pci 0000:10:00.0: BAR 0: no space for [mem size 0x10000000]
    pci 0000:10:00.0: BAR 0: failed to assign [mem size 0x10000000]
    pci 0000:10:00.0: BAR 8: assigned [mem 0xb4200000-0xb43fffff]
    pci 0000:11:00.0: BAR 0: assigned [mem 0xb4200000-0xb43fffff]
    pci 0000:10:00.0: PCI bridge to [bus 11]
    pci 0000:10:00.0:   bridge window [mem 0xb4200000-0xb43fffff]
    Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
    console [ttyS0] disabled
    60300000.uart: ttyS0 at MMIO 0x60300000 (irq = 12, base_baud = 12500000) is a 16550A
    console [ttyS0] enabled
    console [ttyS0] enabled
    bootconsole [uart0] disabled
    bootconsole [uart0] disabled
    msm_serial: driver initialized
    Unable to detect cache hierarcy from DT for CPU 0
    loop: module loaded
    at24 1-0052: 16384 byte 24c128 EEPROM, writable, 64 bytes/write
    Loading iSCSI transport class v2.0-870.
    rdac: device handler registered
    hp_sw: device handler registered
    emc: device handler registered
    alua: device handler registered

Answer 1

With CONFIG_ZONE_DMA enabled, as it is by default on arm64, CMA is restricted to physical memory that devices with 32-bit addressing capabilities can access. Therefore if there isn't enough free RAM below the 4GB boundary to find a contiguous 512MB block (with sufficient alignment) then it's going to fail, regardless of how much RAM exists at higher addresses. That's going to depend on the exact physical memory map of the machine, where the kernel is loaded, what reservations the bootloader has made, etc.

By way of comparison, I could happily boot one of my dev boards with cma=1024M - that one has 2GB of RAM below 4GB, with the kernel loaded near the bottom and only a small firmware reservation near the top, and evidently managed to find a big enough space between the two. The same kernel with ZONE_DMA disabled happily allocated 4GB of CMA (which can only have come out of the upper 6GB block), but I didn't look closely at how badly broken all the peripherals were with the possibility of being given unreachable DMA addresses.