Wrapping gob decoder

Question

I'm storing encrypted gobs in a k-v database and was hoping to have a convenience function in a package which takes a key and an interface, it decrypts the stored value, decodes it into the passed interface and the consumer code can hum along without having to know anything about the data being stored.

I can't for the world figure out why I fail to write to the pointer value using gob, when it works with the json encoder. Dumbed-down the code looks like this:

type Foo struct {
    A string
    B string
}

func marshal(i interface{}) ([]byte, error) {
    var indexBuffer bytes.Buffer
    encoder := gob.NewEncoder(&indexBuffer)
    err := encoder.Encode(&i)
    return indexBuffer.Bytes(), err
}

func unmarshal(data []byte, e interface{}) error {
    buf := bytes.NewBuffer(data)
    dec := gob.NewDecoder(buf)
    err := dec.Decode(&e)
    fmt.Println("Unmarshal", e)
    return err
}

func marshalJ(i interface{}) ([]byte, error) {
    return json.Marshal(i)
}

func unmarshalJ(data []byte, e interface{}) error {
    return json.Unmarshal(data, e)
}

func main() {
    foo := Foo{"Hello", "world!"}
    gob.Register(Foo{})
    data, err := marshal(foo)
    fmt.Println("got", len(data), err)
    var bar Foo
    err = unmarshal(data, &bar)
    fmt.Println("Main err", err)
    fmt.Println("Main", bar)
    fmt.Println("-------------------------")
    data, err = marshalJ(foo)
    fmt.Println("got J", len(data), err)
    err = unmarshalJ(data, &bar)
    fmt.Println("Main J err", err)
    fmt.Println("Main J", bar)
}

(play link)

The unmarshalJ behaves like I expect, that is in Main my bar variable is a copy of foo since unmarshalJ receives a pointer to bar and passes it on.

The unmarshal function on the other hand have the parameter changed locally, but the change isn't made to the bar in Main. Internally gob.Decode passes on reflect.ValueOf, and I have always tried to avoid using reflect so I don't really understand what that does.

My two questions would be:

1. Is it possible to have a wrapper like I want and have the caller's value changed like it works with the json encoder?
2. More of a bonus question because this makes me feel I'm not fully grasping pointers in go: How can the value of unmarshal.e be changed, but Main.bar not be changed when unmarshal.e is a pointer to Main.bar? Another thing which confuse me here is that inside unmarshal I have to pass &e to Decode or it fails with want struct type main.Foo; got non-struct.

Answer 1

There are two issues, 1st is that you give unmarshal a pointer, and then make another pointer inside unmarshal itself, so you end up passing **Foo to the gob decoder.

The second issue is that you take pointers of the interface{}'es inside the function. This somehow effects how the data is encoded. Everything works if you pass in pointers to the functions and don't modify the variables inside the functions.

The fixed code looks like this, playground link:

type Foo struct {
    A string
    B string
}

func marshal(i interface{}) ([]byte, error) {
    var indexBuffer bytes.Buffer
    encoder := gob.NewEncoder(&indexBuffer)
    err := encoder.Encode(i)
    return indexBuffer.Bytes(), err
}

func unmarshal(data []byte, e interface{}) error {
    buf := bytes.NewBuffer(data)
    dec := gob.NewDecoder(buf)
    err := dec.Decode(e)
    fmt.Println("Unmarshal", e)
    return err
}

func marshalJ(i interface{}) ([]byte, error) {
    return json.Marshal(i)
}

func unmarshalJ(data []byte, e interface{}) error {
    return json.Unmarshal(data, e)
}

func main() {
    foo := Foo{"Hello", "world!"}
    gob.Register(Foo{})
    data, err := marshal(&foo)
    fmt.Println("got", len(data), err)
    var bar Foo
    err = unmarshal(data, &bar)
    fmt.Println("Main err", err)
    fmt.Println("Main", bar)
    fmt.Println("-------------------------")
    data, err = marshalJ(foo)
    fmt.Println("got J", len(data), err)
    err = unmarshalJ(data, &bar)
    fmt.Println("Main J err", err)
    fmt.Println("Main J", bar)
}

---

Edit: as response to the comment.

Preventing issues like this is sometimes hard, the root of the issue in my opinion is the use of interface{} which throws away type info, nothing we can do about this unfortunately(other than making an explicit decoder function for each type). The second "issue" is that gob just ignores the fact that types mismatch without error, thus not giving us any indication about what we are doing wrong.

What we can do on the decoding side is to add more strict type checking. We can ask the decoder to just put the decoded value in a interface{} and then check if the decoded type matches the type of e:

type Foo struct {
    A string
    B string
}

func marshal(i interface{}) ([]byte, error) {
    var indexBuffer bytes.Buffer
    encoder := gob.NewEncoder(&indexBuffer)
    err := encoder.Encode(&i)
    return indexBuffer.Bytes(), err
}

func unmarshal(data []byte, e interface{}) (err error) {
    buf := bytes.NewBuffer(data)
    dec := gob.NewDecoder(buf)

    eVal := reflect.ValueOf(e)
    eType := eVal.Type()
    if eVal.Kind() != reflect.Ptr {
        return errors.New("e must be a pointer")
    }

    var u interface{}
    err = dec.Decode(&u)
    uVal := reflect.ValueOf(u)
    uType := uVal.Type()
    if eType.Elem() != uType {
        return fmt.Errorf("decoded type '%s' and underlying type of e '%s' not the same", uType.String(), eType.Elem())
    }

    eVal.Elem().Set(uVal)

    return err
}

func main() {
    foo := Foo{"Hello", "world!"}
    gob.Register(Foo{})
    data, err := marshal(foo)
    fmt.Println("got", len(data), err)
    var bar Foo
    var invalid interface{} = bar
    err = unmarshal(data, &invalid)
    fmt.Println("Main err", err)
    fmt.Println("Main", bar)
    err = unmarshal(data, &bar)
    fmt.Println("Main err", err)
    fmt.Println("Main", bar)
}

Outputs:

got 61 <nil>
Main err decoded type 'main.Foo' and underlying type of e 'interface {}' not the same
Main { }
Main err <nil>
Main {Hello world!}