Description
Hello, I'm new to GRPC (learning 5 days now).
My goal is to generate python SDK for Gitlab Gitaly from published protos.
I started with default protoc but quickly found out the problem with sys.path and everything else mentioned in motivation.
Then I found this project which gave me hope.
At first, I tried the latest release version 1.2.5 which lead NoImplementedError -> Client Streaming.
Then I installed the latest master 5f7e4d58efa8aeb7af28c9ed89e4870548a1ce62 .
This let me process some files but not others.
Gitaly proto structure looks like this:
├── blob.proto
├── cleanup.proto
├── commit.proto
├── conflicts.proto
├── diff.proto
├── errors.proto
├── hook.proto
├── internal.proto
├── lint.proto
├── namespace.proto
├── objectpool.proto
├── operations.proto
├── praefect.proto
├── ref.proto
├── remote.proto
├── repository-service.proto
├── server.proto
├── shared.proto
├── smarthttp.proto
├── ssh.proto
├── transaction.proto
└── wiki.proto
Most of the files depend on shared.proto and lint.proto.
Here is what I tried:
Default protoc:
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_out=${PROTO_OUTPUT}/normal \
--grpc_python_out=${PROTO_OUTPUT}/normal \
${PROTO_PATH}/ssh.proto
Result are 2 files that import lint_pb2 and shared_pb2 which are nowhere to be found... great
ssh_pb2.py
ssh_pb2_grpc.py
Then I tried the wildcard:
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_out=${PROTO_OUTPUT}/normal \
--grpc_python_out=${PROTO_OUTPUT}/normal \
${PROTO_PATH}/*.proto
This created all relevant files but was not really usable due to the broken sys.path.
betterprotoc.
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_betterproto_out=${PROTO_OUTPUT}/better \
${PROTO_PATH}/ssh.proto
Bug #1
The result is the generated package that does not respect the input filenames so everything ended up in __init__.py that was thousands of lines long.
Actual result:
better
├── gitaly
│ └── __init__.py
└── __init__.py
Expected result:
better
├── gitaly
│ ├── __init__.py
│ ├── shared.py
│ ├── lint.py
│ └── ssh.py
└── __init__.py
Bug #2 failure with some files
I also tried to use the wildcard *.proto
python -m grpc_tools.protoc \
--proto_path=${PROTO_PATH} \
--python_betterproto_out=${PROTO_OUTPUT}/better \
${PROTO_PATH}/*.proto
But this resulted in a StopIteration I was not able to debug. What I realised was that this may be a problem only with some files.
I was able to pin down which files have caused this error:
transaction.proto
wiki.proto
repository-service.proto
ref.proto
diff.proto
commit.proto
Traceback shows something about enum type but that's unlikely because the other files that work also contain that type.
Traceback (most recent call last):
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/bin/protoc-gen-python_betterproto", line 8, in <module>
sys.exit(main())
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/main.py", line 32, in main
response = generate_code(request)
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/parser.py", line 107, in generate_code
read_protobuf_service(service, index, output_package)
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/parser.py", line 191, in read_protobuf_service
ServiceMethodCompiler(
File "<string>", line 7, in __init__
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/models.py", line 683, in __post_init__
self.mutable_default_args # ensure this is called before rendering
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/models.py", line 720, in mutable_default_args
and f.default_value_string != "None"
File "/home/user/cache/pypoetry/virtualenvs/gitaly-client-rXVxS8FM-py3.9/lib/python3.9/site-packages/betterproto/plugin/models.py", line 486, in default_value_string
enum = next(
StopIteration
--python_betterproto_out: protoc-gen-python_betterproto: Plugin failed with status code 1.
It would be very nice if betterproto can respect the initial *.proto structure because right now it seems to only read package IIUC.
Please let me know if I can help you with further debugging or this example is good enough.
my environment:
[tool.poetry.dependencies]
python = "~3.9"
grpcio = "^1.44.0"
grpcio-tools = "^1.44.0"
betterproto = {git = "https://github.com/danielgtaylor/python-betterproto.git", rev = "5f7e4d58efa8aeb7af28c9ed89e4870548a1ce62"}
[tool.poetry.dev-dependencies]
pytest = "^5.2"
ptpython = "^3.0.20"
types-protobuf = ">=0.1.14"
betterproto = {extras = ["compiler"], git = "https://github.com/danielgtaylor/python-betterproto.git", rev = "5f7e4d58efa8aeb7af28c9ed89e4870548a1ce62"}