Parameterized lifetime causing trouble in chains of function

Question

playground link

I'm having trouble trouble understanding why this code can't compile:

struct Ctx<'a> {
    n: u64,
    phantom: std::marker::PhantomData<&'a u8>,
}

fn process<'a, 'b: 'a>(ctx: &'b mut Ctx<'a>) {
    ctx.n += 1;
}

fn main() {
    let mut ctx = Ctx { n: 0, phantom: std::marker::PhantomData };
    let ref_ctx = &mut ctx;
    process(ref_ctx);
    process(ref_ctx);
}

The code is silly, but the error is the same as on the "real" code:

   Compiling playground v0.0.1 (/playground)
error[E0499]: cannot borrow `*ref_ctx` as mutable more than once at a time
  --> src/main.rs:14:13
   |
13 |     process(ref_ctx);
   |             ------- first mutable borrow occurs here
14 |     process(ref_ctx);
   |             ^^^^^^^
   |             |
   |             second mutable borrow occurs here
   |             first borrow later used here

For more information about this error, try `rustc --explain E0499`.
error: could not compile `playground` due to previous error

If change the process signature to

fn process(ctx: &mut Ctx<'a>) {

Then it compiles, but I need the lifetimes annotations because in practice Ctx contains references that needs to be annotated. I feel like the problem is 'b: 'a because is could be telling that the borrow MUST outlive the references in Ctx, but it feels weird that just calling a function that returns nothing would extend the borrow,

That feels like a beginner's question, but it definitely has me stumped.

edit:

The code above is minimal because I wanted to avoid explaining the whole context (pun intended), but it might be necessary to find a practical solution. Sorry, that's going to be a bit long.

The real Ctx is here, it's struct that wraps a few Varnish C pointers and gets passed to plugins so they can read it and possibly change it, notably manipulating HTTP objects.

An important part of Ctx is the WS contained in it. It's a workspace that can return non-overlapping slices of writable data that will be deleted automatically once a task has been completed. For the compiler, the best we can tell it is "any slice allocated in the WS will survive for as long as the Ctx wil live".

WS can be used in two ways:

• either directly by a plugin to request a slice it'll fill in
• or by the boilerplate that's currently python-generated to convert Varnish types into C types. Typically, the plugin returns a String and the boilerplate will allocate a WS slice and copy the data for storage. And example of generated code looks like this:

// the C-matching function that needs to exist for the plugin frame work to pick it up
unsafe extern "C" fn vmod_c_unset_hdr(vrt_ctx: * mut varnish_sys::vrt_ctx, arg1: varnish_sys::VCL_STRING) {
    // create the rust Ctx from the pointer
    let mut _ctx = Ctx::new(vrt_ctx);
    // call the plugin function
    match vmod::unset_hdr(
        &mut _ctx,
        &*arg1.into_rust()
    ).into_result() { // make sure we have a Result by wrapping lone values
            Err(ref e) => { _ctx.fail(e); },    // yell if we didn't got and Err
            Ok(v) => v.into_vcl(&mut _ctx.ws),  // convert the Ok value into a C type, possibly by copying it into the workspace
        }
}

Now, so far, the code at the commit I shared works (hurray), but I get into trouble when I try to implement WS_ReserveAll and WS_Release: basically, grab all the free space you can, write what you need, and then tell Varnish how much you used so the rest is reclaimed and can be used by future calls. Of course, you can only WS_ReserveAll() once before you need to WS_Release so that felt like a perfect job for Rust.

I though I'd create a new type ReservedBuf, by adding this to the code:

impl<'a> struct WS<'a> {
// some lines omitted
    pub fn reserve(&'a mut self) -> ReservedBuf<'a> {
        let wsp = unsafe { self.raw.as_mut().unwrap() };
        assert_eq!(wsp.magic, varnish_sys::WS_MAGIC);

        unsafe {
            let sz = varnish_sys::WS_ReserveAll(wsp) as usize;

            let buf = from_raw_parts_mut(wsp.f as *mut u8, sz as usize);
            ReservedBuf {
                buf,
                wsp: self.raw,
                b: wsp.f as *const u8,
                len: 0,
            }
        }
    }

}

pub struct ReservedBuf<'a> {
    pub buf: &'a mut [u8],
    wsp: *mut varnish_sys::ws,
    b: *const u8,
    len: usize,
}

impl<'a> ReservedBuf<'a> {
    pub fn seal(mut self, sz: usize) -> &'a [u8] {
        unsafe {
            self.len = std::cmp::min(sz, self.buf.as_ptr().add(sz).offset_from(self.b) as usize);
            from_raw_parts_mut(self.buf.as_mut_ptr(), self.len)
        }
    }
}

impl<'a> Drop for ReservedBuf<'a> {
    fn drop(&mut self) {
        unsafe {
            let wsp = self.wsp.as_mut().unwrap();
            assert_eq!(wsp.magic, varnish_sys::WS_MAGIC);
            varnish_sys::WS_Release(wsp, self.len as u32);
        }
    }
}

And it appears to work well enough in the limited tests I ran. My plugin function can be this for example:

pub fn ws_reserve<'a, 'b: 'a>(ctx: &'b mut Ctx<'a>, s: &str) -> Result<varnish_sys::VCL_STRING, String> {
    let mut rbuf = ctx.ws.reserve();
    let s_buf = s.as_bytes();
    let vcl_string = rbuf.buf.as_ptr() as *const i8;
    rbuf.buf.write(s_buf);
    rbuf.buf.write(b" ");
    rbuf.buf.write(s_buf);
    rbuf.buf.write(b" ");
    rbuf.buf.write(s_buf);
    rbuf.buf.write(b"\0");
    rbuf.seal(0);
    Ok(vcl_string)
}

but I need the lifetime annotations, otherwise I get:

  --> src/vmod.rs:21:27
   |
20 | pub fn ws_reserve(ctx: &mut Ctx, s: &str) -> Result<varnish_sys::VCL_STRING, String> {
   |                        --------
   |                        |
   |                        these two types are declared with different lifetimes...
21 |     let mut rbuf = ctx.ws.reserve();
   |                           ^^^^^^^ ...but data from `ctx` flows into `ctx` here

but with the lifetime annotations, the generated-code calling ws_reserve complains:

unsafe extern "C" fn vmod_c_ws_reserve(vrt_ctx: * mut varnish_sys::vrt_ctx, arg1: varnish_sys::VCL_STRING) -> varnish_sys::VCL_STRING {
    let mut _ctx = Ctx::new(vrt_ctx);
    match vmod::ws_reserve(
        &mut _ctx,
        &*arg1.into_rust()
    ).into_result() {
            Err(ref e) => { _ctx.fail(e); ptr::null() },
            Ok(v) => v.into_vcl(&mut _ctx.ws),
        }
}

error[E0499]: cannot borrow `_ctx` as mutable more than once at a time
  --> /home/gquintard/project/varnish-rs/vmod_test/target/debug/build/vmod_example-7ff3d57b642f2bfa/out/generated.rs:69:29
   |
66 |         &mut _ctx,
   |         --------- first mutable borrow occurs here
...
69 |             Err(ref e) => { _ctx.fail(e); ptr::null() },
   |                             ^^^^
   |                             |
   |                             second mutable borrow occurs here
   |                             first borrow later used here

error[E0499]: cannot borrow `_ctx.ws` as mutable more than once at a time
  --> /home/gquintard/project/varnish-rs/vmod_test/target/debug/build/vmod_example-7ff3d57b642f2bfa/out/generated.rs:70:33
   |
66 |         &mut _ctx,
   |         --------- first mutable borrow occurs here
...
70 |             Ok(v) => v.into_vcl(&mut _ctx.ws),
   |                                 ^^^^^^^^^^^^
   |                                 |
   |                                 second mutable borrow occurs here
   |                                 first borrow later used here

For more information about this error, try `rustc --explain E0499`.
error: could not compile `vmod_example` due to 2 previous errors

I guess I sort of understand the compiler qualms about the issues, and I might have painted myself into a corner, but I don't see how to solve this.

Answer 1

When we take a closer look at your code we see you write this:

fn process<'a, 'b: 'a>(ctx: &'b mut Ctx<'a>) {

This 'b: 'a reads as 'b outlives 'a. So you're basically writing a method which borrows ctx longer than what ctx is holding internally with 'a.

With that knowledge it seems very logical why this won't work.

let mut ctx = Ctx { n: 0, phantom: std::marker::PhantomData };
let ref_ctx = &mut ctx;
process(ref_ctx); // Borrow ref_ctx for at least the lifetime of `Ctx::<'a>`
process(ref_ctx); // Can't borrow it again since `'a` has not expired yet

What you're trying to do is probably this:

fn process<'a: 'b, 'b>(ctx: &'b mut Ctx<'a>) {

Borrow ctx for the lifetime 'b which is less long than 'a

If that's the case you can just write this:

fn process<'a>(ctx: &mut Ctx<'a>) {

Or even this:

fn process(ctx: &mut Ctx) {

If this isn't what you're trying to do I hope it still makes sense that using the constraint 'b: 'a creates an impossible borrow.