CODEWITHSUNDEEP
goyaml
Virmundi
Virmundi

Reputation: 2631

Dynamically Unmarshaling yaml in golang

Edit: Adding problem context

I'm making a migration tool for the ArangoDB database. Don't know what that is? Checkout https://www.arangodb.com. Now like most migration tools, mine needs to create Collections, which are logically equivalent to SQL Tables. It needs to create or remove indexes, collections, graphs, and other DB entities as well as execute arbitrary AQL (SQL like algebra for ArangoDB).

To support this the user creates a series of migration files, each saying what to do with that step. So step 1 might say "Create Collection Users" and step 67 might say "Delete Collection Users -- since we renamed it". I want to use YAML because I was explicitly asked by the ArangoDB team why I didn't. Golang seems like a better choice than my existing Java implementation due to the binary distribution not requiring Java.

All that said, I have several different types of structs stored in different yaml files. I can tell which struct is in which file by reading the first line of the yaml. I want to be as DRY as possible with reading the file.

I tried to have a factory method that creates the proper struct to from the file in this code. We can assume that all instances of pickT will return a valid implementation of the Migration interface.

func pickT(contents []byte) (Migration, error) {
    s := string(contents)
    switch {
    case collection.MatchString(s):
        return new(Collection), nil
    default:
        return nil, errors.New("Can't determine YAML type")
    }
}

func toStruct(childPath string) Migration {
    contents := open(childPath)

    t, err := pickT(contents)
    if err != nil {
        log.Fatal(err)
    }
    //c := Collection{}
    err = yaml.UnmarshalStrict(contents, &t)
    if err != nil {
        log.Fatal(err)
    }
    return t
}

This code does not work. Yaml kicks out the error below.

--- FAIL: TestLoadFromPath (0.00s)
panic: reflect.Set: value of type map[interface {}]interface {} is not assignable to type main.Migration [recovered]
    panic: reflect.Set: value of type map[interface {}]interface {} is not assignable to type main.Migration [recovered]
    panic: reflect.Set: value of type map[interface {}]interface {} is not assignable to type main.Migration

goroutine 5 [running]:
testing.tRunner.func1(0xc4201060f0)
    /usr/local/go/src/testing/testing.go:711 +0x2d2
panic(0x68ade0, 0xc420010cf0)
    /usr/local/go/src/runtime/panic.go:491 +0x283
gopkg.in/yaml%2ev2.handleErr(0xc420057b18)
    /home/jdavenpo/go/src/gopkg.in/yaml.v2/yaml.go:164 +0x9f
panic(0x68ade0, 0xc420010cf0)
    /usr/local/go/src/runtime/panic.go:491 +0x283
reflect.Value.assignTo(0x69e740, 0xc42007d230, 0x15, 0x6f30e4, 0xb, 0x6a45a0, 0xc420010cd0, 0x69e740, 0xc42007d230, 0x15)
    /usr/local/go/src/reflect/value.go:2192 +0x3a6
reflect.Value.Set(0x6a45a0, 0xc420010cd0, 0x194, 0x69e740, 0xc42007d230, 0x15)
    /usr/local/go/src/reflect/value.go:1357 +0xa4
gopkg.in/yaml%2ev2.(*decoder).mapping(0xc420060900, 0xc4200644e0, 0x6a45a0, 0xc420010cd0, 0x194, 0x6a45a0)
    /home/jdavenpo/go/src/gopkg.in/yaml.v2/decode.go:522 +0x8e2
gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc420060900, 0xc4200644e0, 0x6a45a0, 0xc420010cd0, 0x194, 0xc4200644e0)
    /home/jdavenpo/go/src/gopkg.in/yaml.v2/decode.go:292 +0x136
gopkg.in/yaml%2ev2.(*decoder).document(0xc420060900, 0xc420064480, 0x6a45a0, 0xc420010cd0, 0x194, 0xc4200608c0)
    /home/jdavenpo/go/src/gopkg.in/yaml.v2/decode.go:304 +0x80
gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc420060900, 0xc420064480, 0x6a45a0, 0xc420010cd0, 0x194, 0x194)
    /home/jdavenpo/go/src/gopkg.in/yaml.v2/decode.go:280 +0x1d2
gopkg.in/yaml%2ev2.unmarshal(0xc42010c000, 0x3b, 0x23b, 0x67c4a0, 0xc420010cd0, 0x1, 0x0, 0x0)
    /home/jdavenpo/go/src/gopkg.in/yaml.v2/yaml.go:101 +0x289
gopkg.in/yaml%2ev2.UnmarshalStrict(0xc42010c000, 0x3b, 0x23b, 0x67c4a0, 0xc420010cd0, 0x0, 0x0)
    /home/jdavenpo/go/src/gopkg.in/yaml.v2/yaml.go:87 +0x58
github.com/deusdat/arangomigo.toStruct(0xc420014360, 0x26, 0x0, 0x0)
    /home/jdavenpo/go/src/github.com/deusdat/arangomigo/migrations.go:102 +0x295
github.com/deusdat/arangomigo.loadFrom(0x6f8059, 0x1a, 0x0, 0x0, 0x0)
    /home/jdavenpo/go/src/github.com/deusdat/arangomigo/migrations.go:67 +0x4d6
github.com/deusdat/arangomigo.TestLoadFromPath(0xc4201060f0)
    /home/jdavenpo/go/src/github.com/deusdat/arangomigo/migrations_test.go:11 +0x48
testing.tRunner(0xc4201060f0, 0x705950)
    /usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:789 +0x2de

To the best of my understanding, Go type erases the interface, at least functionally. Is there any way to dynamically pick the type and then have Yaml convert it out? I know that I can do this manually by repeating the same logic over and over in different functions like the one below, but this seems...terrible. 

func fromCollectionYaml(content string) Migration {
    c := Collection{}
    err := yaml.Unmarshal(content, &c)
    if err != nil {
        log.Fatal(err)
    }
    return c
}

Edit: Adds the Collection struct and Yaml

type: collection
Action: create
Name: Users
Journalsize: 10

type Collection struct {
    Name           string
    Action         Action
    ShardKeys      []string
    JournalSize    int
    NumberOfShards int
    WaitForSync    bool
    AllowUserKeys  bool
    Volatile       bool
    Compactable    bool
}

func (this Collection) migrate(action Action, db *arangolite.Database) error {
    return nil
}

In the near future I need to add structs for Indexes. Graphs and other ArangoDB related features. Each Yaml file needs to go into a struct that produces and migration, which is just a instance of an interfaces with the one function currently shown along with the struct.

Upvotes: 2

Views: 2349

Answers (1)

infiniteLearner
infiniteLearner

Reputation: 4189

I found answer to this question on reddit , to help future readers I am providing content below with link to the post where I found it.

The solution, I believe, is to simply pass t, not &t to Unmarshal.

Rationale: You need to pass a pointer to a struct which can take the intended data as a dynamic value in the interface{} to Unmarshal. From pickT, you return new(Collection), which is such a pointer. But then, you first store it in another interface type (Migration) and then take the pointer of that. Meaning, yaml will a) see that it gets passed a pointer and b) try to store the value in the pointee. But the pointee is of interface type (and that interface value will then contain the actual struct, but yaml doesn't do this additional redirection).

If you instead pass t (which has interface type) it will first be converted to interface{} and then passed on - so the argument will directly contain a *Collection, as intended.

The difference here is, that if you convert/assign one interface type to another, it won't get re-wrapped, but the dynamic value in one value will just be transferred to another. But if you store a non-interface type (like a pointer to an interface), it will get re-wrapped.

It's not easily explained, I'm afraid. You can see the effect here too: https://play.golang.org/p/ia181c_Pwp (note, that Printf also takes an interface{})

https://www.reddit.com/r/golang/comments/7k0kpv/please_help_with_dynamically_unmarshaling_yaml_in/

Upvotes: 1

Related Questions