How to build a REST API with gRPC and get the best of two worlds

Build a gRPC Service with REST support from an OpenAPI description.

As part of this Google Summer of Code project I had the opportunity to work with Tim and Noah (two SWE at Google) to build a tool called gnostic-grpc. This tool converts an OpenAPI v3.0 API description into a description of a gRPC service that can be used to implement that API using gRPC-JSON transcoding. gRPC services are described with the Protocol Buffers language (.proto).

With the converted description of a gRPC service you can generate your API in any programming language of your choice.

In case you need an introduction to gRPC, OpenAPI, and Protocol Buffers I recommend this excellent blog post by Tim.

The following tutorials assume that you have a basic understanding of Protocol Buffers and the protoc compiler set up.

Tutorial — gRPC gateway plugin

This tutorial has seven steps:

1. Generate a gRPC service (.proto) from an OpenAPI description.

2. Generate server-side support code for the gRPC service.

3. Implement the server logic.

4. Generate the descriptor set for the envoy proxy.

5. Set up an envoy proxy that provides HTTP transcoding.

6. Run the gRPC server.

7. Test your API with curl and a gRPC client.

At the end of this tutorial our architecture looks like this:

Prerequisites:

Let’s get the plugin first:

go get -u github.com/googleapis/gnostic-grpc

Ideally, you follow the steps while you are inside of the examples/end-to-end directory.

Now we get the other dependencies we need:

go get -u github.com/googleapis/gnostic
go get -u github.com/golang/protobuf/protoc-gen-go
go get -u google.golang.org/grpc

gnostic is a command line tool that converts OpenAPI descriptions to equivalent Protocol Buffer representations. gnostic-grpc is a plugin for gnostic.

protoc-gen-go is a plugin for the protoc compiler to generate Go source code from Protocol Buffer definitions.

At last, we get grpc.

For simplicity we create a temporary environment variable inside of our current terminal:

export ANNOTATIONS="third-party/googleapis"

First step:

Inside of the examples/end-to-end directory execute the plugin to obtain a Protocol Buffer definition from a given OpenAPI description (bookstore.yaml):

gnostic --grpc-out=. bookstore.yaml

This command generates the file bookstore.proto. You might see a bunch of warnings on your terminal. Those warnings describe which parts of the OpenAPI description are not reflected inside of the output file.

Under the hood the command triggers gnostic which generates a binary Protocol Buffer representation of the OpenAPI description. This representation is then used to build a FileDescriptorSet. The FileDescriptorSet essentially represents the .proto file we want to generate. Then we use protoreflect to generate the output file

Second step:

Now we generate the gRPC stubs:

protoc --proto_path=. --proto_path=${ANNOTATIONS} --go_out=plugins=grpc:bookstore bookstore.proto

This command generates the file bookstore/bookstore.pb.go.

— proto_path=. tells the protoc compiler to look inside the current directory for .proto files.

— proto_path=${ANNOTATIONS} tells the protoc compiler to look inside the third-party/googleapis directory for .proto files. The annotations are necessary to define the gRPC/REST mapping.

— go_out=plugins=grpc:bookstore tells the protoc compiler to generate Go source code using the grpc plugin. The output should be generated into a directory called bookstore.

bookstore.proto is our input file for the compiler.

Third step:

We provide a sample implementation of the server logic inside bookstore/server.go. All the data structures that were generated previously are used within that server.

Fourth step:

Given the bookstore.proto file we can generate the FileDescriptorSet with protoc:

protoc --proto_path=${ANNOTATIONS} --proto_path=. --include_imports --include_source_info --descriptor_set_out=envoy-proxy/proto.pb bookstore.proto

This generates the file envoy-proxy/proto.pb. As explained in the first step of the other tutorial a FileDescriptorSet represents a .proto file in binary format. Essentially it is equal to bookstore.descr we see in the figure “gnostic and gnostic-grpc” in step 1.

Fifth step:

The file envoy-proxy/envoy.yaml contains an envoy configuration with a gRPC-JSON transcoder. According to the configuration, port 51051 proxies gRPC requests to a gRPC server running on localhost:50051 and uses the gRPC-JSON transcoder filter to provide the RESTful JSON mapping. I.e.: you can either make gRPC or RESTful JSON requests to localhost:51051.

Use docker to get the envoy image:

docker pull envoyproxy/envoy-dev:bcc66c6b74c365d1d2834cfe15b847ae13be0eb6

The file envoy-proxy/Dockerfile uses the envoy image we just pulled as base image and copies envoy.yaml and proto.pb to the filesystem of the docker container:

FROM envoyproxy/envoy dev:bcc66c6b74c365d1d2834cfe15b847ae13be0eb6
COPY envoy.yaml /etc/envoy/envoy.yaml
COPY proto.pb /tmp/envoy/proto.pb

Build a docker image from the docker file:

docker build -t envoy:v1 envoy-proxy

Run the docker container with the created image on port 51051:

docker run -d --name envoy -p 9901:9901 -p 51051:51051 envoy:v1

Sixth step:

Run the gRPC server on port 50051:

go run main.go

Seventh step:

Now let’s test our gRPC/REST API:

Inside of a new terminal we create a shelve:

curl -X POST \
http://localhost:51051/shelves \
-H 'Content-Type: application/json' \
-d '{
    "name": "Books I need to read",
    "theme": "Non-fiction"
}'

To check whether that worked we get all existing shelves:

curl -X GET http://localhost:51051/shelves

Now we create a book for the shelve with the id 1 (the shelve we just created):

curl -X POST \
http://localhost:51051/shelves/1/books \
-H 'Content-Type: application/json' \
-d '{
    "author": "Hans Rosling",
    "name": "Factfulness",
    "title": "Factfulness: Ten Reasons We'\''re wrong about the world - and Why Things Are Better Than You Think"
}'

And to list all books for the shelve with id 1 we can call:

curl -X GET http://localhost:51051/shelves/1/books

Ok, so it looks like our REST API is working. What about a gRPC client?

Inside grpc-client/client.go we provide a sample implementation of the gRPC client. The client prints all themes of your shelves:

client := bookstore.NewBookstoreClient(conn) 
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second) res, err := client.ListShelves(ctx, &empty.Empty{}) 
if res != nil {
 fmt.Println("The themes of your shelves:")
 for _, shelf := range res.Ok.ListShelvesResponse.Shelves {
   fmt.Println(shelf.Theme)
 }
}

To run the client, execute the following command:

go run grpc-client/client.go

Notice that the gRPC client also calls the envoy proxy (port 51051) and not the gRPC server directly (port 50051). However, inside the gRPC client you can also change the port to 50051.

Final word:

To sum it up: we see that we can generate a lot of code that we typically have to code manually which is more error prone. Feel free to contribute to gnostic-grpc.