How to access user space function arguments in bpf?

Question

I am trying to instrument a user space nginx function by using libbpf. I am able to attach a uprobe it, and print pid, tid and so on from the probe. However, I am having great issues whenever I try to parse function argument data. I have been able to do this with bpftrace but am unable to do so with libbpf. My question is, how to properly access and print arguments of the user space function I want to trace?

nginx.bpf.c

#include "ngx_http.h"
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>

char LICENSE[] SEC("license") = "Dual BSD/GPL";

SEC("uprobe//usr/sbin/nginx:ngx_http_finalize_request")
int handle_ngx_http_finalize_request(struct ngx_http_request_s* r, ngx_int_t rc)
{
    u_char *s_ptr;
    u_char str[128];
    int err;

    err = bpf_probe_read_user(&s_ptr, sizeof(s_ptr), &r->request_line.data);
    if (!s_ptr || err < 0) {
        bpf_printk("Error %d\n", err);
        return -2;
    }

    bpf_probe_read_user_str(str, sizeof(str), &s_ptr);

    bpf_printk("String: %s\n", str);

    return 0;
}

Whenever I try to parse the function arguments the bpf_probe_read_user returns error -14. When I try to use bpf_core_read the verifier rejects the code with following error.

❯ sudo ./nginx
libbpf: loading object 'nginx_bpf' from buffer
libbpf: elf: section(3) uprobe//usr/sbin/nginx:ngx_http_finalize_request, size 280, link 0, flags 6, type=1
libbpf: sec 'uprobe//usr/sbin/nginx:ngx_http_finalize_request': found program 'handle_ngx_http_finalize_request' at insn offset 0 (0 bytes), code size 35 insns (280 bytes)
libbpf: elf: section(4) .reluprobe//usr/sbin/nginx:ngx_http_finalize_request, size 32, link 12, flags 40, type=9
libbpf: elf: section(5) license, size 13, link 0, flags 3, type=1
libbpf: license of nginx_bpf is Dual BSD/GPL
libbpf: elf: section(6) .rodata, size 22, link 0, flags 2, type=1
libbpf: elf: section(7) .BTF, size 12720, link 0, flags 0, type=1
libbpf: elf: section(9) .BTF.ext, size 252, link 0, flags 0, type=1
libbpf: elf: section(12) .symtab, size 240, link 1, flags 0, type=2
libbpf: looking for externs among 10 symbols...
libbpf: collected 0 externs total
libbpf: map 'nginx_bp.rodata' (global data): at sec_idx 6, offset 0, flags 80.
libbpf: map 0 is "nginx_bp.rodata"
libbpf: sec '.reluprobe//usr/sbin/nginx:ngx_http_finalize_request': collecting relocation for section(3) 'uprobe//usr/sbin/nginx:ngx_http_finalize_request'
libbpf: sec '.reluprobe//usr/sbin/nginx:ngx_http_finalize_request': relo #0: insn #13 against '.rodata'
libbpf: prog 'handle_ngx_http_finalize_request': found data map 0 (nginx_bp.rodata, sec 6, off 0) for insn 13
libbpf: sec '.reluprobe//usr/sbin/nginx:ngx_http_finalize_request': relo #1: insn #28 against '.rodata'
libbpf: prog 'handle_ngx_http_finalize_request': found data map 0 (nginx_bp.rodata, sec 6, off 0) for insn 28
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: map 'nginx_bp.rodata': created successfully, fd=4
libbpf: sec 'uprobe//usr/sbin/nginx:ngx_http_finalize_request': found 1 CO-RE relocations
libbpf: prog 'handle_ngx_http_finalize_request': relo #0: <byte_off> [2] typedef ngx_http_request_t.request_line.data (0:21:1 @ offset 992)
libbpf: prog 'handle_ngx_http_finalize_request': relo #0: no matching targets found
libbpf: prog 'handle_ngx_http_finalize_request': relo #0: substituting insn #1 w/ invalid insn
libbpf: prog 'handle_ngx_http_finalize_request': BPF program load failed: Invalid argument
libbpf: prog 'handle_ngx_http_finalize_request': -- BEGIN PROG LOAD LOG --
R1 type=ctx expected=fp
; int handle_ngx_http_finalize_request(ngx_http_request_t* r, ngx_int_t rc)
0: (bf) r3 = r1
1: <invalid CO-RE relocation>
failed to resolve CO-RE relocation <byte_off> [2] typedef ngx_http_request_t.request_line.data (0:21:1 @ offset 992)
processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'handle_ngx_http_finalize_request': failed to load: -22
libbpf: failed to load object 'nginx_bpf'
libbpf: failed to load BPF skeleton 'nginx_bpf': -22
Failed to open and load BPF skeleton

Here is bpftrace code that works.

nginx.bt

uprobe:/usr/sbin/nginx:ngx_http_finalize_request
{
    $req = (struct ngx_http_request_s*)arg0;
    printf("Request Line: %s\n", str($req->request_line.data));
}

So I would like to parse data and do some custom logic depending on it.

Here are the ngx_http_request_s and the ngx_str_t structs for the curious.

EDIT

---

User space code:

#include <stdio.h>
#include <unistd.h>
#include <sys/resource.h>
#include <bpf/libbpf.h>
#include "nginx.skel.h"

static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
{
    return vfprintf(stderr, format, args);
}

int main(int argc, char **argv)
{
    struct nginx_bpf *skel;
    int err;

    libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
    libbpf_set_print(libbpf_print_fn);

    skel = nginx_bpf__open_and_load();
    if (!skel) {
        fprintf(stderr, "Failed to open and load BPF skeleton\n");
        return 1;
    }

    err = nginx_bpf__attach(skel);
    if (err) {
            fprintf(stderr, "Failed to auto-attach BPF skeleton: %d\n", err);
            goto cleanup;
    }

    printf("Successfully started!\n");

    for (;;) {
        sleep(1);
    }

cleanup:
    nginx_bpf__destroy(skel);
    return -err;
}

Answer 1

You are actually not defining BPF program correctly. You should do something like the below. Note that you also shouldn't use CO-RE variants of bpf_core_read() or BPF_CORE_READ(), use BPF_PROBE_READ_USER() instead:

SEC("uprobe//usr/sbin/nginx:ngx_http_finalize_request")
int BPF_KPROBE(handle_ngx_http_finalize_request,
               struct ngx_http_request_s* r, ngx_int_t rc)
{
    u_char *s_ptr;
    u_char str[128];
    int err;

    /* you can access rc directly now, btw */

    s_ptr = BPF_PROBE_READ_USER(r, request_line.data);
    /* note no dereferencing of s_ptr above */
    bpf_probe_read_user_str(str, sizeof(str), s_ptr); 

    bpf_printk("String: %s\n", str);

    return 0;

You don't have to use BPF_PROBE_READ_USER() macro, you can do the same with just bpf_probe_read_user() like you did in your example. BPF_PROBE_READ_USER() will be especially handy if you need to follow few levels of pointers, though.

But there is no CO-RE for user-space types, it's only for kernel types, because kernel provides BTF information to allow relocating offsets properly.