What is the relationship between proto package and actual directory structure

Question

I have read official docs & tutorial online about protobuf but I still don't quite understand the relationship between the package name & import path in proto definition and actual directory structure.

To make my question clear, let's say this is my project structure:

root
├── project
│   └── protos
│       ├── common
│       │   └── common.proto
│       └── custom
│           ├── app.proto
│           └── util
│               └── util.proto           

common.proto

syntax = "proto3";

// how should I decide the package for this file based on my project structure?
package project.protos.common;

message Common {
    // ...
}

app.proto, which references common.proto (in another separate directory) and util.proto (in subdirectory)

syntax = "proto3";

package project.protos.custom;

import "project/protos/common/common.proto";
import "project/protos/custom/util/util.proto";
import "google/protobuf/timestamp.proto";

message App {
    project.protos.common.Common common = 3;
    project.protos.custom.util.Util util = 4;
}

util.proto

syntax = "proto3";

package project.protos.custom.util;

option java_multiple_files = true;

message Util {
    // ...
}

After doing several trials, it turns out that if I run the protoc command at the root directory, using the command:

protoc -I . --java_out=. project/protos/**/*.proto

then imports are working correctly and every proto file is compiled with no error. However, let's say if I run the protoc not at root directory, but at subdirectory project, using similar command:

protoc -I . --java_out=. protos/**/*.proto

Compilation would then fail with following error:

project/protos/common/common.proto: File not found.
project/protos/custom/util/util.proto: File not found.
protos/custom/app.proto:6:1: Import "project/protos/common/common.proto" was not found or had errors.
protos/custom/app.proto:8:1: Import "project/protos/custom/util/util.proto" was not found or had errors.
protos/custom/app.proto:19:5: "project.protos.common.Common" seems to be defined in "protos/common/common.proto", which is not imported by "protos/custom/app.proto".  To use it here, please add the necessary import.
protos/custom/app.proto:20:5: "project.protos.custom.util.Util" is not defined.

As I'm working with Java for most of times, I'm currently thinking the way that how package & import works in protobuf, with respect to the actual directory structure, is similar to how package & import works in Java, with respect to classpath.

That is, if I want to compile a proto with package specified to be "project.protos.common", then I should run the protoc command on the parent directory of project dir, where project/protos/common is a valid path.

Similarly, if I am importing a proto in another proto file with the import path set to be "project/protos/custom/util/util.proto", then I should also make sure that from the directory when I run protoc command, path project/protos/custom/util/util.proto is present.

I'm currently guessing whatever path we feed into -I option, protoc will treat them in the same way as Java classpath or python search path. Upon importing, protoc will look for the imported file on every specified -I path, following the relative path that is indicated by import statement.

However, after searching online for similar questions and reading some articles, now I have the impression that package and import in protobuf are actually much more flexible than I thought them to be. Is my understanding wrong here?

Answer 1

I'm going to answer indirectly:

1. Package names correspond to directories
2. One Package may span multiple files in a single directory
3. (But) Individual files must import Protos even if in the same Package
4. It's useful to namespace|scope packages by some org id (e.g. google for timestamp)
5. A hierarchy (e.g. google.protobuf manifest as google/protobuf/...) of related Packages is independent of its file system location.
6. Although the Package hierarchy is fixed, there's flexibility in 'placement' when generating language-specific bindings.

I'll try to provide examples of each:

Using protoc's include directory as an example, the Package google.protobuf clearly identifies this as a google originated Package representing protobuf "stuff".

The Package must be represented in a directory as google/protobuf but it's location in the system is arbitrary and without dependencies (see --proto_path).

Files in the package e.g. api.proto must import e.g. type.proto (using the full namespace path (i.e. google/protobuf/type.proto) even though they share a Package.

When running protoc .... foo.proto, if foo.proto imports google/protobug/api.proto, protoc needs some means of finding the Package and so protoc --proto_path=path/to/google is required (although if protoc is in the path, the include directory need not be added on Linux). All imports must be similarly --proto_path'd.

Where protoc outputs files and (more importantly) the namespace used to produce the ouput, has some flexibility thanks to option [langauge]_package flags in the proto files.

I'm unfamiliar with Java, but in the case of Golang, I can use option go_package to help me specify a Golang Module name (github.com/[org]/[repo]) to ensure that the generated files are placed in the repo so that they will be correctly imported by other Golang modules.