grpcio-tools/grpc_tools.protoc How do i use the custom package path

Question

Hi i have a question on how to use the custom package path when calling grpc_tools.protoc My project structure is the following

C:\temp\test
    caller.py
    --proto
      --foo.proto
      --bar.proto
    --generated

and here some example protos. The foo.proto importing the bar.proto. All are at the same folder level.

syntax = "proto3";
package mydummy;
import "bar.proto";
import "google/protobuf/empty.proto";

service FooService{
    rpc Foo(google.protobuf.Empty) returns (FooResponse);
}


message FooResponse {
    oneof result {
        int32 mask = 1;
        FailureMessage failure = 2;
    }
}

syntax = "proto3";
package mydummy;


message FailureMessage {
    string message = 1;
}

When I now generate the *.py with following command and custom package path -Igenerated=C:\temp\test\proto

python -m grpc_tools.protoc -Igenerated=C:\temp\test\proto --python_out=. --pyi_out=. --grpc_python_out=. C:\temp\test\proto\*.proto

result

bar.proto: File not found.
generated/foo.proto:3:1: Import "bar.proto" was not found or had errors.
generated/foo.proto:13:9: "mydummy.FailureMessage" seems to be defined in "generated/bar.proto", which is not imported by "generated/foo.proto".  To use it here, please add the necessary import.

If I don't use the custom package path I have the issue with the relative imports in the generated files when setting the e.g. python_out to --python_out=./generated Am I using the package path -I wrong? The official documentation is not so helpful at this point.

Greetings

Answer 1

Try using the absolute path for generated when passing it to grpcio_tools.protoc.

python -m grpc_tools.protoc -I"C:\temp\test\generated"="C:\temp\test\proto" --python_out=. --pyi_out=. --grpc_python_out=. C:\temp\test\proto\*.proto

That's what works for me. Although generated will also need to be a Python package, with an __init__.py file for the files to be importable, but at least they will be able to find each other now.

Answer 2

There are a couple of issues:

1. package dummy

Though not required, it is good practice to reflect package paths in the folder path of your protobuf sources.

In this case, one (!) of your --proto_path (-I) values will need to reference ${PWD}/proto because that is the root folder containing your protos.

Because your protos are in package dummy, I'd encourage (again not required but good practice), you to relocate them to a sub-folder dummy:

./proto
└── mydummy
    ├── bar.proto
    └── foo.proto

Because you've relocated your protos, you'll need to revise foo.proto's import for bar.proto:

syntax = "proto3";

package mydummy;

import "mydummy/bar.proto";            // <--- Now includes "package"
import "google/protobuf/empty.proto";

service FooService{
    rpc Foo(google.protobuf.Empty) returns (FooResponse);
}

message FooResponse {
    oneof result {
        int32 mask = 1;
        FailureMessage failure = 2;
    }
}

For examples of this in practice, se Google's Well-known Types:

Where e.g. google.protobuf.Timestamp is defined in ${PROTO}/google/protobuf/timestamp.proto

/path/to/protoc/include
└── google
    └── protobuf
        ├── ...
        ├── timestamp.proto
        └── ...

2. --proto_path (-I)

The proto_path flag must be used for each protobuf source path that's imported by your protobuf sources with the exception of the aforementioned Google Well-known Types which are included by default.

This is why you don't need to include ${PROTOC}/include as one of the --proto_path flags but you must include a reference to your proto folder.

3. generated

The generated folder is an output folder. It doesn't not contain protobuf sources and should not form part of the --proto_path (-I) flag folders.

That said, here's your protoc command:

PROTOPATH="${PWD}/proto"

python \
-m grpc_tools.protoc \
--proto_path=${PROTOPATH} \
--python_out=${PWD}/generated \
--pyi_out=${PWD}/generated \
--grpc_python_out=${PWD}/generated \
${PROTOPATH}/mydummy/*.proto

I've used PROTOPATH in an attempt to clarify how proto_path works.

Every *.proto that you reference must be contained with one of the proto_path values that you reference.

Because your proto_path root is ${PWD}/proto, you must then reference protobuf sources within it by their folder (now equal to package) path, i.e. ${PROTOPATH}/mydummy/*.proto

4. Visual Studio Code

If you're using Visual Studio Code, there's a protocol buffer plugin: vscode-proto3.

You can configure this plugin to reflect your proto_path's by adding a (workspace-specific) ${PWD}/.vscode/settings.json file. In your case, to reflect your use of the proto folder:

{
    "protoc": {
        "options": [
            "--proto_path=${workspaceFolder}/proto"
        ]
    }
}