CODEWITHSUNDEEP
memory-managementlinux-kernelhuge-pages
Mark
Mark

Reputation: 6474

error setting nr_hugepages via SYSFS

I have 8G of physical memory, Fedora20, and configured kernel parameters to allocate two 1G hugepages at boot-up time by passing the following parameters to the kernel:

default_hugepagesz=1G hugepagesz=1G hugepages=2

HugeTLBFS is auto mounted:

% mount | grep ^huge
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
%

Upon reboot, all looks good, and I see the kernel has allocated the required pages:

% dmesg | grep HugeTLB
HugeTLB registered 1 GB page size, pre-allocated 2 pages
% grep -E ^"(Mem|Huge)" /proc/meminfo
MemTotal:        8137732 kB
MemFree:         5359672 kB
MemAvailable:    5707656 kB
HugePages_Total:       2
HugePages_Free:        2
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
%

As you see the amount of free memory should allow me to increase number of hugepages, however I fail to do so:

% echo 3 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
-bash: echo: write error: Invalid argument

or

% echo 3 > /proc/sys/vm/nr_hugepages
-bash: echo: write error: Invalid argument
%

Also decreasing the number of pages fail. What am I doing wrong?

Upvotes: 3

Views: 6243

Answers (2)

Mark
Mark

Reputation: 6474

I think I have found the reason of this behaviour. The kernel documentation for "hugepagesz=" parameter in https://www.kernel.org/doc/Documentation/kernel-parameters.txt says that "1GB pages can only be allocated at boot time using hugepages= and not freed afterwards" I think this is why I can't decrease number of pages.

Also, the code in mm/hugetlb.c doesn't allow this operation:

#if defined(CONFIG_CMA) && defined(CONFIG_X86_64)
...
static inline bool gigantic_page_supported(void) { return true; }
#else
static inline bool gigantic_page_supported(void) { return false; }
...
#endif
...
static int hugetlb_sysctl_handler_common(...)
{
  ...
  if (write && hstate_is_gigantic(h) && !gigantic_page_supported())
          return -EINVAL;
  ...
}

So, unless CONFIG_CMA is defined (and in the default kernel configuration shipped with Fedora 20 this option is disabled), the kernel will always return EINVAL.

Upvotes: 4

askb
askb

Reputation: 6768

Looks like you may require superuser permissions for the command. Alternatively, you could try the below step if you want to make changes persistent on your system.

First its required to mount the hugetlbfs for traditional hugepages to work.

mkdir /hugetlbfs
mount -t hugetlbfs none /hugetlbfs

Note: IA-64 supports - 4KiB, 2MiB and 4MiB pages Note: x86_64 supports - 4KiB, 2MiB, 4MiB or 1GiB pages

Next, depending on your requirement, edit /etc/sysctl.conf file and specify the number of hugepages in the nr_hugepages: 

vm.nr_hughpages=2

Now run the command sysctl -p for the configuration changes to take effect. Note: if your requirement of number of huge pages are large and no contiguous free blocks available, then its recommended to restart the system after the above changes.

To check if the number of hugepages have been allocated, use cat /proc/meminfo | grep Huge 

HugePages_Total:    2
HugePages_Free:     2
HugePages_Rsvd:     0
Hugepagesize:       2048 kB

Upvotes: 1

Related Questions