deciphering "invalid mem access map_ptr" on BPF program upload

Question

I'm writing a custom application that reads C code, calls LLVM to generate BPF byte code from that C code, then relocates any bpf map symbols and uploads it to the kernel. I can successfully upload and run programs that don't use BPF maps, but as soon as I relocate a program to use a BPF map I get the following error:

invalid mem access 'map_ptr'

Details follow:

The following input is given to LLVM:

// Placeholder values for user-requested maps
void *a;

#include <linux/ptrace.h>
#include <uapi/linux/bpf.h>
#include "bpf_helpers.h"

int kprobe__blk_start_request(struct pt_regs *ctx) {
  long rq = PT_REGS_PARM1(ctx);
  u64 val = bpf_ktime_get_ns();
  bpf_map_update_elem(a, &rq, &val, BPF_ANY);
  return 0;
}

which generates the following bytecode

Found function: kprobe__blk_start_request
  size: 120 bytes
  addr: 0
relocate: a @ 32

/tmp/bpf7990/bpf.o: file format ELF64-BPF

Disassembly of section .text:
kprobe__blk_start_request:
       0:   79 11 70 00 00 00 00 00     r1 = *(u64 *)(r1 + 112)
       1:   7b 1a f8 ff 00 00 00 00     *(u64 *)(r10 - 8) = r1
       2:   85 00 00 00 05 00 00 00     call 5
       3:   7b 0a f0 ff 00 00 00 00     *(u64 *)(r10 - 16) = r0
       4:   18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00     r1 = 0ll
       6:   79 11 00 00 00 00 00 00     r1 = *(u64 *)(r1 + 0)
       7:   bf a2 00 00 00 00 00 00     r2 = r10
       8:   07 02 00 00 f8 ff ff ff     r2 += -8
       9:   bf a3 00 00 00 00 00 00     r3 = r10
      10:   07 03 00 00 f0 ff ff ff     r3 += -16
      11:   b7 04 00 00 00 00 00 00     r4 = 0
      12:   85 00 00 00 02 00 00 00     call 2
      13:   b7 00 00 00 00 00 00 00     r0 = 0
      14:   95 00 00 00 00 00 00 00     exit

and the "a" symbol gets relocated in instruction #4 as follows:

Relocating instruction 4
    from code:0x18 dst_reg:1 src_reg:0 off:0x0 imm:0x0
      to code:0x18 dst_reg:1 src_reg:1 off:0x0 imm:0x4
kprobe__blk_start_request:
       0:   79 11 70 00 00 00 00 00 
       1:   7b 1a f8 ff 00 00 00 00 
       2:   85 00 00 00 05 00 00 00 
       3:   7b 0a f0 ff 00 00 00 00 
       4:   18 11 00 00 04 00 00 00 
       5:   00 00 00 00 00 00 00 00 
       6:   79 11 00 00 00 00 00 00 
       7:   bf a2 00 00 00 00 00 00 
       8:   07 02 00 00 f8 ff ff ff 
       9:   bf a3 00 00 00 00 00 00 
      10:   07 03 00 00 f0 ff ff ff 
      11:   b7 04 00 00 00 00 00 00 
      12:   85 00 00 00 02 00 00 00 
      13:   b7 00 00 00 00 00 00 00 
      14:   95 00 00 00 00 00 00 00

The "04" in instruction #4 is the FD to the map for symbol "a". After calling BPF_PROG_LOAD I get the following in the error log from the kernel.

Failed to load kprobe__blk_start_request BPF code
0: R1=ctx(id=0,off=0,imm=0) R10=fp0
0: (79) r1 = *(u64 *)(r1 +112)
1: R1=inv(id=0) R10=fp0
1: (7b) *(u64 *)(r10 -8) = r1
2: R1=inv(id=0) R10=fp0
2: (85) call bpf_ktime_get_ns#5
3: R0=inv(id=0) R10=fp0
3: (7b) *(u64 *)(r10 -16) = r0
4: R0=inv(id=0) R10=fp0
4: (18) r1 = 0xffff88042be0b000
6: R0=inv(id=0) R1=map_ptr(id=0,off=0,ks=8,vs=8) R10=fp0
6: (79) r1 = *(u64 *)(r1 +0)
R1 invalid mem access 'map_ptr'

Errno: 13 (Permission denied)

I'm having trouble deciphering this error log. What is the kernel trying to tell me?

Answer 1

Error explanation

invalid mem access 'map_ptr' means you're trying to read from an invalid memory location, in particular one pointed to by a map pointer.

Indeed, referring to your bytecode:

   4:   18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00     r1 = 0ll
   6:   79 11 00 00 00 00 00 00     r1 = *(u64 *)(r1 + 0)

you first load an immediate value inside r1, then read the memory location pointed to by r1. However, the kernel verifier identifies BPF_LD_IMM instructions as map pointer loading. So, it tags r1 with the map_ptr type. The verifier then rejects instruction #6 because it tries to read the memory location pointed to by the map pointer (you're only supposed to pass it to map helpers, not use it otherwise in BPF programs).

Basically, instruction #6 is both invalid and unneeded. Without it, your program should pass the verifier.

---

Likely fix

I don't have your relocation code so I can't reproduce, but I'd guess this invalid bytecode as to do with how you declare the map. If you take a look at BPF samples in the Linux kernel, you'll see that maps are usually declared as global structure to which the first map helper argument points:

// Placeholder values for user-requested maps
struct bpf_map_def a = {
    .type = BPF_MAP_TYPE_ARRAY,
    .key_size = sizeof(u32),
    .value_size = sizeof(int),
    .max_entries = 1024,
};

#include <linux/ptrace.h>
#include <uapi/linux/bpf.h>
#include "bpf_helpers.h"

int kprobe__blk_start_request(struct pt_regs *ctx) {
  long rq = PT_REGS_PARM1(ctx);
  u64 val = bpf_ktime_get_ns();
  bpf_map_update_elem(&a, &rq, &val, BPF_ANY);
  return 0;
}