I finally got time to put my focus on my most popular open source project grpc-mate, an enterprise ready micro service project base on gRPC. the java based project was completed about 1.5 years ago, now it's time to develop a python based project to demostrate the best practice in python. I find it's a good way for me to learn a new things. before I could write it out, I will have to understand it very well.

python-grpc

dependency setup

to setup a virtual env for python development, please checkout this article. grpc depends on protobuffer to serialize and deserialize message,so we need both grpc and protobuf as dependencies in the project pipenv install grpcio protobuf, we also need grpcio-tools to generate grpc python stubs, it will generate both client and server stubs in python, so we will need to issue command pipenv install grpcio-tools

generate grpc python stubs

we could re-use the protobuffer definitions in grpc-mate project, to use it in grpc-mate-python project we could create a folder under grpc-mate named protobuffers and create symlink of grpc_mate and google into this folder so that we will have only one place to manage all the grpc api definition.

we could define package structure of protobuf for different language. like example below, we could define the java package and go_package as different package name in it's own language,

syntax = "proto3";
option java_package = "io.datanerd.generated.es";
option java_multiple_files = true;
import "grpc_mate/product_common.proto";
import "google/api/annotations.proto";
option go_package = "datanerd";

but python is doing other way, it's respecting the protobuf package structure, the python package will be the same as the protobuf package structure, in our case our user defined protobuf will generate python package in package grpc_mate and the google protobuf will generate python files in package google, to generate the python stubs, we could issue command like below

rm -fR grpc_mate/* && \
	rm -fR google/* && \
	python  -m grpc_tools.protoc -Iprotobuffers --python_out=. --grpc_python_out=.  protobuffers/grpc_mate/*.proto protobuffers/google/api/*.proto && \
	touch grpc_mate/__init__.py
	touch google/__init__.py
	touch google/api/__init__.py

we are touching __init__.py for each package to make it backwards compatible, in python 3.6 this is optional. it is also a good idea to place the auto generated python code in it's own package so that we are confident to re-generate them.

implement the grpc service

grpcio-tools already generate the server stubs in xxxx_pb2_grpc to implement GreeeterSerevice, we could create a class which extends from grpc_mate.helloworld_pb2_grpc.GreeterServicer and override SayHello function like below, isn't it easy? we could access passed in parameter from request

import logging
import grpc_mate.helloworld_pb2
import grpc_mate.helloworld_pb2_grpc

logger = logging.getLogger(__name__)

class GreeterServicer(grpc_mate.helloworld_pb2_grpc.GreeterServicer):

    def SayHello(self, request, context):
        logger.debug(f"get request {request.name}")
        return grpc_mate.helloworld_pb2.HelloReply(message=f"hello {request.name}")

add grpc service and start grpc server

  • to create a grpc server we will need to pass grpc server a thread pool to use for rpc handlers
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
  • add the service to the grpc server
    grpc_mate.helloworld_pb2_grpc.add_GreeterServicer_to_server(GreeterServicer(), server)
  • tell grpc server which port it could listen to, if port not specified grpc will choosee a random port and return it
    server.add_insecure_port('[::]:8080')
  • start the grpc server
    server.start()
  • let server to wait there to accpet incomming call
    server.wait_for_termination()

to put everything together,the code is like below

def serve():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    grpc_mate.helloworld_pb2_grpc.add_GreeterServicer_to_server(GreeterServicer(), server)
    server.add_insecure_port('[::]:8080')
    server.start()
    logger.debug('grpc server started at port 8080')
    server.wait_for_termination()


if __name__ == '__main__':
    serve()

test call to grpc server

I will write another article to describe how to do grpc's unit test, now I just use BloomRPC to the the call manually

  • load the protobuf into BloomRPC
  • open Greeter Service
  • click SayHello node in the left panel
  • input server address as localhost:8080
  • in the left panel, input parameter as "name":"Ivan"
  • click "go" button in middle of the panels
  • get the message as "message": "hello Ivan"

bloomrpc

write grpc client to call the server

def integration_test_SayHello():
    from grpc_mate.helloworld_pb2_grpc import GreeterStub
    channel = grpc.insecure_channel('localhost:8080')
    stub = GreeterStub(channel)
    hello_request = HelloRequest(name='local')
    response = stub.SayHello(hello_request)
    assert response.message == f'hello {hello_request.name}'