CODEWITHSUNDEEP
linuxlinux-kernelsystemtap
user2366736
user2366736

Reputation: 31

systemtap global variable allocation failed

I want to use systemtap for extracting details of my linux production server. my systemtap script is 

global bt;
global quit = 0

probe begin {
    printf("start profiling...\n")
}
probe timer.profile {
    if (pid() == target()) {
        if (!quit) 
        {
            bt[backtrace(), ubacktrace()] <<< 1
        } 
        else 
        {

            foreach ([sys, usr] in bt- limit 1000) 
            {
                print_stack(sys)
                print_ustack(usr)
                printf("\t%d\n", @count(bt[sys, usr]))
            }
            exit()
        }
    }
}

probe timer.s(20) {
    quit = 1
}

When I start run this script with command 

sudo stap --ldd -d $program_name --all-modules                  \
    -D MAXMAPENTRIES=10240 -D MAXACTION=20000 -D MAXTRACE=40    \
    -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=40 -x $program_pid     \
    profile.stp  --vp 00001 > profile.out

It fails, and prints following error:

ERROR: error allocating hash
ERROR: global variable 'bt' allocation failed
WARNING: /usr/bin/staprun exited with status: 1

my production server memory info is

             total       used       free     shared    buffers     cached
Mem:         16008      15639        368          0         80       3090
-/+ buffers/cache:      12468       3539

I think it is enough, because in my test server, there is only 2G memory, and the systemtap script runs well for another server

Upvotes: 1

Views: 757

Answers (1)

myaut
myaut

Reputation: 11504

Unfortunately, this is intended behavior, see my discussion here: https://sourceware.org/ml/systemtap/2015-q1/msg00033.html

The problem is that SystemTap allocates associative arrays at once (to prevent allocation failures in future) and on per-cpu basis (to prevent locking), which means that bt will require (2 * MAXSTRINGLEN + sizeof(statistic)) * MAXMAPENTRIES * NR_CPU =~ 2 Gb if NR_CPU == 128.

Reduce MAXSTRINGLEN (which is set to 4k in your case) or size of bt array:

global bt[128];

Upvotes: 1

Related Questions