CODEWITHSUNDEEP
clinux-kernelelfhuge-pages
xiaodong
xiaodong

Reputation: 1012

usage of hugepages for text and data segments

I am exploring the possibility to back the text and data segment with hugepages following the guide in https://github.com/libhugetlbfs/libhugetlbfs/blob/master/HOWTO.

I have relinked the application as suggested by adding "-B/usr/share/libhugetlbfs -Wl,--hugetlbfs-align" and started the application with "hugectl --text --data --heap --bss /path/to/my/application".

But I am not very sure how to verify whether the data and text segments are indeed copied to the files on the hugetlbfs filesystem.

Checking the /proc/{pid}/maps, it could be seen that the hugepages are used for heap but not for text and data segments as the first two address ranges are mapped to the application but not the hugepage file system.

Is my understanding correct? Actually I suspect my conclusion that hugepages are used for heap from /proc/{pid}/maps is also incorrect.

How should I verify whether the data and text segments are backed in the hugepages? I know that data and text segments will be copied to hugetlbfs filesystem if successful but how to verify it?

Thanks!

output of /proc/{pid}/maps

00400000-00d2c000 r-xp 00000000 fd:02 46153351 /path/to/my/application

00f2b000-00fa3000 rw-p 0092b000 fd:02 46153351 /path/to/my/application

00fa3000-00fbb000 rw-p 00000000 00:00 0

02a0c000-02a2d000 rw-p 00000000 00:00 0  [heap]

40000000-80000000 rw-p 00000000 00:15 2476090 /dev/hugepages-1G/libhugetlbfs.tmp.nS7exn (deleted)

Upvotes: 3

Views: 2077

Answers (3)

the8472
the8472

Reputation: 43042

The easiest way to get text sections backed by huge pages on current linux kernels is to compile it with the experimental CONFIG_READ_ONLY_THP_FOR_FS=y option and setting the THP policy to always.

Some distros ship kernels with that set, you can check with zgrep THP_FOR_FS /proc/config.gz

Upvotes: 1

Jacek Tomaka
Jacek Tomaka

Reputation: 422

If you set HUGETLB_DEBUG=1 variable, it will tell you whole lot of useful information. One of them is this:

INFO: Segment 2's aligned memsz is too small: 0x864 < 0xffffffffffffffff

If it is successful it looks like:

libhugetlbfs [zupa:154297]: INFO: Segment 0 (phdr 2): 0x400000-0x400864 (filesz=0x864) (prot = 0x5)

libhugetlbfs [zupa:154297]: INFO: Segment 1 (phdr 3): 0x600868-0x600af8 (filesz=0x27c) (prot = 0x3)

libhugetlbfs [zupa:154297]: DEBUG: Total memsz = 0xaf4, memsz of largest segment = 0x864

libhugetlbfs [zupa:154297]: INFO: libhugetlbfs version: 2.16 (modified)

libhugetlbfs [zupa:154951]: INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x864 bytes and 0 extra bytes from 0x400000...done

libhugetlbfs [zupa:154297]: INFO: Prepare succeeded

libhugetlbfs [zupa:154952]: INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x27c bytes and 0 extra bytes from 0x600868...done

libhugetlbfs [zupa:154297]: INFO: Prepare succeeded

Upvotes: 1

ceph3us
ceph3us

Reputation: 7474

check 

   /proc/$pid/numa_maps

contains information about each memory area used by a given process allowing--among other information--the determination of which nodes were used for the pages.

for formar see http://linux.die.net/man/5/numa_maps

Upvotes: 2

Related Questions