How to fix this python import shadowing problem?

Question

Problem Overview

I'm facing an import shadowing problem with a python 3.12 project that uses google protobuf.

Below is the structure of my project.

Project
├── proto (generated by protoc)
│   ├── google
│   │   └── api
│   │       └── ...
│   └── my_proto (depends on google.api protos)
│       └── ...
└── main.py

Virtualenv
└── lib
    └── python3.12
        └── site-packages
            └── google
                └── protobuf

I'm using a virtual environment to manage dependencies and protoc to generate python code. The problem is that in some of the generated code proto/my_proto, it attempts to do both import google.protobuf (which is the protobuf package) and the generated proto code module import google.api.xxxx. One of the imports is guaranteed to fail because the proto/google module shadows the one in the virtual environment. How can I fix this problem?

Minimal Repro

Get the Googleapi protos

clone this repo here: https://github.com/googleapis/googleapis

Create a directory with the following content

.
├── project
│   ├── proto
│   │   └── __init__.py
│   ├── __init__.py
│   └── __main__.py
├── proto
│   └── my_proto.proto
├── build.sh
└── pyproject.toml

__main__.py

from .proto.my_proto_pb2 import *

def main():
    pass

my_proto.proto

syntax = "proto3";
import "google/api/annotations.proto";

message MyMessage {
    string example = 1;
}

pyproject.toml

[tool.poetry]
name = "project"
version = "0.1.0"
description = ""
authors = []
readme = "README.md"

[tool.poetry.scripts]
main = "project.__main__:main"

[tool.poetry.dependencies]
python = "^3.12"
grpcio-tools = "1.66.1"
grpcio = "1.66.1"
protobuf = "5.27.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

build.sh

proto_folder="./proto"
google_proto_folder="<dir containg this repo: https://github.com/googleapis/googleapis>"

poetry lock
poetry install
poetry run python -m grpc_tools.protoc \
    -I"$proto_folder" \
    -I"$google_proto_folder/googleapis" \
    --experimental_allow_proto3_optional \
    --python_out="./project/proto" \
    --grpc_python_out="./project/proto" \
    $google_proto_folder/googleapis/google/api/annotations.proto \
    $proto_folder/my_proto.proto

poetry install

Build and Run

change the working directory to the project root and run the following:

./build.sh
poetry run main

the following error message will be shown:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/tongweid/Documents/repos/avsim-service/_local/repro/project/project/__main__.py", line 1, in <module>
    from .proto.my_proto_pb2 import *
  File "/home/tongweid/Documents/repos/avsim-service/_local/repro/project/project/proto/my_proto_pb2.py", line 25, in <module>
    from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
ModuleNotFoundError: No module named 'google.api'

I suspect this is because google.protobuf shadows the generated google.api.

Answer 1

You're using the wrong technique to import an adjacent package from within your existing package. In Python versions since 2.6 or so (see PEP 328), all imports are assumed to be absolute unless explicitly specified otherwise.

Rather than using google.api to refer to proto.google.api, you need to either use the full name, or an explicitly relative name, starting with a leading dot: .google.api.

Try:

from .google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2

(Though, wow, that name is aweful, why are you doing that to yourself? With just from . import google.api you can then reference google.api.annotiations_pb2 directly, without needing a mangled name with loads of underscores.)