ARM GCC heap not fully used

Question

I am setting up my Cortex-M4 platform to use heap memory and encountering some issues. I set heap region size to be 512 bytes, and it only allocates 9 bytes. Then I set heap to be 10kB and it can only allocate 362 bytes. Here is my gcc stub:

int _sbrk(int a)
{
    //align a to 4 bytes
    if (a & 3)
    {
        a += (4 - (a & 0x3));
    }

    extern long __heap_start__;
    extern long __heap_end__;
    static char* heap_ptr = (char*)&__heap_start__;

    if (heap_ptr + a < (char*)&__heap_end__)
    {
        int res = (int)heap_ptr;
        heap_ptr += a;
        return res;
    }
    else
    {
        return -1;
    }
}

__heap_start__ and __heap_end__ are correct and their difference show correct region size. I added debug in _sbrk function to see what a argument is passed when this function is called and the values of that argument are like these in each call respectively:

2552
1708
4096

What can I do to make it use full heap memory? And how _sbrk argument is calculated? Basically, what's wrong here?

Building C++ code, using new (std::nothrow).

EDIT If I am using malloc (C style) it allocates 524 bytes and no _sbrk call before main, unlike when using operator new.

arm-none-eabi-g++.exe (GNU Tools for ARM Embedded Processors 6-2017-q2-update) 6.3.1 20170620 (release) [ARM/embedded-6-branch revision 249437]
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

EDIT2 Minimal Complete Verifiable Example
Here is my application code and _sbrk with info printing:

void foo()
{
    while (true)
    {
        uint8_t * byte = new (std::nothrow) uint8_t;
        if (byte)
        {
            DBB("Byte allocated");
            cnt++;
        }
        else
        {
            DBB_ERROR("Allocated %d bytes", cnt);
        }
    }
}

int _sbrk(int a)
{
    //align a to 4 bytes
    if (a & 3)
    {
        a += (4 - (a & 0x3));
    }

    extern long __heap_start__;
    extern long __heap_end__;
    static char* heap_ptr = (char*)&__heap_start__;

    DBB("%d 0x%08X", a, a);
    DBB("0x%08X", heap_ptr);
    DBB("0x%08X", &__heap_start__);
    DBB("0x%08X", &__heap_end__);

    if (heap_ptr + a < (char*)&__heap_end__)
    {
        int res = (int)heap_ptr;
        heap_ptr += a;
        DBB("OK 0x%08X 0x%08X", res, heap_ptr);
        return res;
    }
    else
    {
        DBB("ERROR");
        return -1;
    }
}

And produced output is:

Answer 1

Your output reveals the C++ memory allocation system first asks for 32 bytes and then 132 bytes. It is then able to satisfy nine requests for new uint8_t with that space. Presumably it uses some of the 164 bytes for its internal record-keeping. This may involve keeping link lists or maps of which blocks are allocated, or some other data structure. Also, for efficiency, it likely does not track single-byte allocations but rather provides some minimum block size for each allocation, perhaps 8 or 16 bytes. When it runs out of space it needs, it asks for another 4096 bytes. Your sbrk then fails since this is not available.

The C++ memory allocation system is working as designed. In order to operate, it requires more space than is doled out for individual requests. In order to supply more memory for requests, you must provide more memory in the heap. You cannot expect a one-to-one correspondence, or any simple correspondence, between memory supplied from sbrk to the memory allocation system and memory supplied from the memory allocation system to its clients.

There is no way to tell the C++ memory allocation system to use “full heap memory” to satisfy requests to it. It is required to track dynamic allocations and releases of memory. Since its clients may make diverse size requests and may release them in any order, it needs to be able to track which blocks are currently allocated and which are not—a simple stack will not suffice. Therefore, it must use additional data structures to keep track of memory, and those data structures will consume space. So not all of the heap space can be given to clients; some of it must be used for overhead.

If the memory use of the memory allocation system in your C++ implementation is too inefficient for your purposes, you might replace it with one you write yourself or with third-party software. Any implementation of the memory allocation system makes various trade-offs about speed and block size, and those can be tailored to particular situations and goals.