Large size kmalloc in the linux kernel kmalloc

Question

I am looking at Linux version 4.9.31

And a kmalloc() function of slab and slub

The following is the kmalloc() function of include/linux/slab.h

static __always_inline void *kmalloc(size_t size, gfp_t flags)
{
    if (__builtin_constant_p(size)) {
        if (size > KMALLOC_MAX_CACHE_SIZE)
            return kmalloc_large(size, flags);
#ifndef CONFIG_SLOB
        if (!(flags & GFP_DMA)) {
            int index = kmalloc_index(size);

            if (!index)
                return ZERO_SIZE_PTR;

            return kmem_cache_alloc_trace(kmalloc_caches[index],
                    flags, size);
        }   
#endif
    }   
    return __kmalloc(size, flags);
}

In the above code, kmalloc_large() is called when __builtin_constant_p(size) is true.

First question. What is the relationship between __builtin_constant_p(size) and kmalloc_large()? Should not kmalloc_large() be called in runtime, not compile time?

The following is the __kmalloc() and __do_kmalloc() of mm/slab.c

static __always_inline void *__do_kmalloc(size_t size, gfp_t flags,
                      unsigned long caller)
{
    struct kmem_cache *cachep;
    void *ret;

    cachep = kmalloc_slab(size, flags);
    if (unlikely(ZERO_OR_NULL_PTR(cachep)))
        return cachep;
    ret = slab_alloc(cachep, flags, caller);

    kasan_kmalloc(cachep, ret, size, flags);
    trace_kmalloc(caller, ret, 
              size, cachep->size, flags);

    return ret; 
}

void *__kmalloc(size_t size, gfp_t flags)
{
    return __do_kmalloc(size, flags, _RET_IP_);
}

The following is the __kmalloc() of mm/slub.c

void *__kmalloc(size_t size, gfp_t flags)
{
    struct kmem_cache *s;
    void *ret;

    if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
        return kmalloc_large(size, flags);

    s = kmalloc_slab(size, flags);

    if (unlikely(ZERO_OR_NULL_PTR(s)))
        return s;

    ret = slab_alloc(s, flags, _RET_IP_);

    trace_kmalloc(_RET_IP_, ret, size, s->size, flags);

    kasan_kmalloc(s, ret, size, flags);

    return ret; 
}

Second question. Why do slub __kmalloc() check "size > KMALLOC_MAX_CACHE_SIZE" and call kmalloc_large() at runtime ?

Answer 1

Your two question are actually parts of the single question:

What is __builtin_constant_p(size)?

Operator __builtin_constant_p is gcc-specific extension, which checks whether its argument can be evaluated at compile time. E.g., if you call

p = kmalloc(100, GFP_KERNEL);

then the operator returns true.

But with

size_t size = 100;
p = kmalloc(size, GFP_KERNEL);

the operator returns false*.

By knowing that some function's parameter is known at compile time, one may check it at compile time, and perform some optimizations.

if (__builtin_constant_p(size)) {
    if (size > KMALLOC_MAX_CACHE_SIZE)

While size > KMALLOC_MAX_CACHE_SIZE seems to be runtime-check here, it is actually compile-time check, because outer condition garantees that size is known at compile time. With that knowledge, compiler may optimize out inner branch, if it is false (if the branch true, compiler may optimize out other branches).

E.g.,

p = kmalloc(100000, GFP_KERNEL);

will be compiled into

kmalloc_large(100000, GFP_KERNEL);

and

p = kmalloc(100, GFP_KERNEL);

will be compiled into

__kmalloc(100, GFP_KERNEL);

But

size_t size = 100000;
p = kmalloc(size, GFP_KERNEL);

will be compiled into

size_t size = 100000;
__kmalloc(size, GFP_KERNEL);

because compiler cannot predict the branch at compile time.

Implementation of "fall-back" function __kmalloc checks its parameters anywhere, for the case when compile-time checks cannot be performed.

---

*- in my recent tests compiler actually doesn't try to predict value of size variable which has been assigned directly with a constant. But this may be changed in future gcc versions.