Trying to write shared protobuf definitions

Question

I'm trying to create shared protobuf definitions in a shared go module, but I'm not having much luck. Specifically:
- I want a proto file in module protos that contains shared definitions.

- Other proto files in module protos would reference the shared definitions.
- Module protos-use would reference module protos (via go.mod)
- Module protos-use go code would import and use definitions from module protos
- Both modules are stored in github.com

I'm unable to get the various package names to align. No matter what I use, something fails. Currently, module protos doesn't have any errors, but go mod tidy on module protos-use fails.

Module protos is at https://github.com/rob-woerner/protos
Module protos-use is at https://github.com/rob-woerner/protos-use

I haven't found an example that addresses this scenario. If I simply duplicate the shared proto definitions everywhere, I can get it to work. The most recent error is:
$ go mod tidy
go: finding module for package github.com/rob-woerner/protos/messages
go: finding module for package github.com/rob-woerner/protos/common
github.com/rob-woerner/protos-use/code imports
github.com/rob-woerner/protos/common: module github.com/rob-woerner/protos@latest found (XXX), but does not contain package github.com/rob-woerner/protos/common
github.com/rob-woerner/protos-use/code imports
github.com/rob-woerner/protos/messages: module github.com/rob-woerner/protos@latest found (XXX), but does not contain package github.com/rob-woerner/protos/messages

But I emphasize that I get lots of different errors, depending on what package names I use in the dozen or so places that need package names.

If anyone knows of a working example, or if this is not supported, I'd appreciate it.

Code snippets are elided for compactness

---

Module protos:

go.mod
Makefile
protobuf:
   common.proto
   messages.proto
pb:
   common.pb.go
   messages.pb.go

---

Module protos-use:
go.mod
code:
   project.go

---

Module protos: go.mod:

module protos
go 1.16
require google.golang.org/protobuf v1.27.1

---

protobuf/common.proto:
package common;
option go_package = "github.com/rob-woerner/protos";
enum MyState { LOGGED_OUT = 0; }

---

protobuf/messages.proto:
import "common.proto";
package messages;
option go_package = "github.com/rob-woerner/protos";
message MyRequest { common.MyState state = 1; }

---

Makefile:
protos:
   protoc --proto_path=protobuf --go_out=plugins=grpc:pb --go_opt=module=github.com/rob-woerner/protos protobuf/*.proto

---

Module protos-use:

go.mod:
module github.com/rob-woerner/protos-use
go 1.16
require (
   github.com/rob-woerner/protos latest
)

---

code/project.go:
package code
import (
   "github.com/rob-woerner/protos/common"
   "github.com/rob-woerner/protos/messages"
)
func MyFunc() messages.MyMessage {
   return messages.MyMessage { state: common.MyState_LOGGED_OUT}
}

Answer 1

This is challenging with protobufs and is poorly documented.

Part of the complexity -- I sense -- is that the protobuf maintainers are trying to solve for the myriad runtimes and packages that may be used.

If I understand your problem correctly, with what you have, protos-use is struggling because code/product.go should:

import (
    "github.com/rob-woerner/protos/pb/common" // /pb/
    ...
)

I've got a working mechanism that I'll try to apply to your mechanism. I think the references to common aren't your issue but the overall structure and modules vs. packages imports:

• github.com/foo/protos
• github.com/bar/app

In foo/protos, I have *.proto files in the root (but this is just a convention and need not be the case):

• example.proto

syntax = "proto3";

package baz;

option go_package = "github.com/foo/protos;baz";

service Some {...}

NOTE The ;baz this describe the eventual Go package name (baz) of the generated code. It need not match the the repo name (protos) and I don't want this because (see below), I'm putting the generated code in a non-root directory (/some/baz)

Then:

MODULE="github.com/foo/protos"
protoc \
--include_imports \
--include_source_info \
--proto_path=. \
--descriptor_set_out=foo.pb \
--go_out=./some/baz \
--go_opt=module=${MODULE} \
--go-grpc_out=./some/baz \
--go-grpc_opt=module=${MODULE} \
./*.proto

NOTE This will give us a package name of github.com/foo/protos/some/baz

This will yield:

.
├── example.proto
├── some
│   └── baz
│       ├── example_grpc.pb.go
│       └── example.pb.go
├── go.mod
├── go.sum
└── protoc-3.17.3-linux-x86_64

Then, from bar/app, I have:

go.mod:

module github.com/bar/app

go 1.16

require (
    github.com/foo/protos v0.0.1
    ...
)

And, within that module, I alias the imports:

package main

import (
    ...
    pb "github.com/foo/protos/some/baz"
)

func main() {
    ...
    pb.RegisterSomeServer(grpcServer, ...)

Update

NOTE Responding to the comment thread

common and messages are either in one package or in two.

Let's do two:

common.proto:

syntax = "proto3";

package common;

option go_package = "github.com/rob-woerner/protos/common;common";

enum MyState { LOGGED_OUT = 0; }

messages.proto:

syntax = "proto3";

package messages;

import "common.proto";

option go_package = "github.com/rob-woerner/protos/messages;messages";

message MyRequest { common.MyState state = 1; }

NOTE Because we want 2 packages, we need to give the protos (!) unique package, go_package names ./protos/common;common. Although we could have ./protos/common;freddie, the issue is that the path must be unique to give a unique package.

Then:

MODULE="github.com/rob-woerner/protos"

protoc \
--proto_path=. \
--go_out=./some \
--go_opt=module=${MODULE} \
--go-grpc_out=./some \
--go-grpc_opt=module=${MODULE}

NOTE The use of some here is solely to ensure that the generated code ends up aggregated by some package (some) below the repo(==module) root. You can remove /some if you want the packages directly beneath the module root.

yields:

some
├── common
│   └── common.pb.go
└── messages
    └── messages.pb.go

NOTE some becomes the root package github.com/rob-woerner/protos/some and common.pb.go is in the sub-package github.com/rob-woerner/protos/some/common

So, we can now:

main.go:

package main

import (
    "github.com/rob-woerner/protos/some/common"
    "github.com/rob-woerner/protos/some/messages"
)

func main() {
    rqst := messages.MyRequest{
        state: common.MyState_LOGGED_OUT,
    }
    // do something with `rqst`
}

If you want common and messages to be in the same package, then you can just:

package protos;

option go_package = "github.com/rob-woerner/protos;protos`;

And:

message MyRequest { MyState state = 1; } // Now in same package

And then:

MODULE="github.com/rob-woerner/protos"

protoc \
--proto_path=. \
--go_out=./protos \
--go_opt=module=${MODULE} \
--go-grpc_out=./protos \
--go-grpc_opt=module=${MODULE}

NOTE In this case, we need _out=./protos so that the generated files get put under the correct package (protos

main.go:

package main

import (
    pb "github.com/rob-woerner/protos"
)

func main() {
    rqst := pb.MyRequest{
        state: pb.MyState_LOGGED_OUT,
    }
    // do something with `rqst`
}

Phew! I said it is challenging ;-)