An Introduction to gRPC and Protocol Buffers for Beginners

What is gRPC? Why is it important?

Published in

AWS in Plain English

8 min readNov 12, 2021

What is gRPC?

gRPC is a modern open-source high-performance Remote Procedure Call (RPC) framework that can run in any environment. It was developed by Google and is now part of the Cloud Native Computation Foundation(CNCF).

It was developed by Google in 2015. Most people assume the ‘g’ to mean Google, but it does not. Google keeps on changing the meaning of the letter with each version to the point that they’ve added a README just to define it.

Some of the features of gRPC are:

It is free and open-source.
gRPC is part of the CNCF foundation, just like prominent technologies like Docker and Kubernetes.
It allows defining a request and a response for your RPC(Remote Procedural Calls)
It is built on HTTP/2, which makes it low-latent and fast.
Has seamless integration with auth, logging & monitoring, which gives it a significant edge in cloud-native applications

gRPC is growing fast and is currently used in organizations like Google, Netflix, Square, etc., and in technologies like CoreOS, CockroachDB, etc.

What is an API?

An Application Programmable Interface (API) is a documented way in which external consumers know how to interact with your code. The components of an API are:

Data model: The representation of the data that is to be interacted with. It might be in a human-readable format (JSON, XML, etc.) or a computer-readable format (Binary).
End Point: The location of your API. It also defines the kind of request to be made (GET, POST, PUT, etc.)
Invocation & Error Handling: How will the API call be made? What should be returned in case of a failure/error?
Latency: This is related to the performance of the API. Clients will need to know the delay expected to design consumers for it.
Scalability: The number of requests that can be handled is another important aspect to define the availability and request rate for the API.
Load Balancers: Load balancing is important to make sure your server does not get bombarded by requests/ DDOS attacks.
Authentication, monitoring & Logging: These are all important components that need to be taken care of for the security & measurement of performance of the API.

At its core, an API:
1. Sends a Request (from Client)
2. Receives a Response (via Server)
3. Has some data (in either human-readable or computer understandable format)

gRPC lets you define 4 kinds of API requests in your service:

Unary Requests: A simple RPC where the client sends a request to the server and waits for the response to come back, just like a normal function call. eg. All REST APIs

Server Streaming Request: A server-side streaming RPC where the client sends a request to the server and gets a stream of messages back. The client reads from the returned stream until there are no more messages. eg. Video/Audio streaming platforms

Client Streaming Request: A client-side streaming RPC where the client writes a sequence of messages and sends them to the server, again using a provided stream. Once the client has finished writing the messages, it waits for the server to read them all and return its response. eg. File upload services.

Bidirectional Streaming Request: A bidirectional streaming RPC where both sides send a sequence of messages using a read-write stream. The two streams operate independently, so clients and servers can read and write in whatever order they like. eg. Chatting applications

gRPC Architecture

gRPC can be used for many languages, but they officially support only 12 very popular ones as mentioned on their website. gRPC has been implemented in 3 different languages: Java, Go & C. These are called pure implementations of gRPC.
All other languages (C++, Python, Ruby, Objective C, PHP, C#, etc.) use the gRPC C implementation.
Beneath the core implementation is HTTP/2 which provides gRPC with its unique capabilities like Bidirectional streaming, Multiplexing, Header compression, etc. It also provides additional security through inbuilt protocols.
The underlying technology for it is just plain Unix Domain Sockets and TCP.

What are RPCs?

RPCs stands for Remote Procedural Calls. In distributed Computing, a remote procedure call (RPC) is when a computer program causes a procedure (subroutine) to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction.

In simpler terms, the client call refers to a method in the server as if it were its own. It uses IDL (Interface Definition Language) as a form of contract on functions to be called and on the data type.

RPC is not a new concept. It’s a rather old concept that has been implemented very cleanly in gRPC. Most of the boilerplate & connections have been written in gRPC as a framework which makes it very easy to use.

The gRPC Process

At the core of gRPC, you need to define messages and services using Protocol Buffers. The rest of the gRPC code is auto-generated and we just need to add an implementation for it. The generated .proto file can support over 12 languages (officially). This allows it to be scaled to over a million requests per second.

Protocol Buffers

A Protocol Buffer, aka Protobuf, is the most commonly used IDL (Interface Definition Language) for gRPC. It’s where we store the data and function contracts in the form of a proto file. A simple proto file looks like this:

message Person {
    required string name = 1;
    required int32 id = 2;
    optional string email = 3;
}

It is a contract defined between the server and the client. Any change in the entity structure or request/response needs to be updated both in the server as well as the client. Protocol buffers use computer-readable formats for sending/receiving data and hence are much faster than traditional REST modules.

Some of the features of protocol buffers are:

Language agnostic: Protocol Buffers are language agnostic as proto files act as intermediaries for servers and clients. proto files can be generated for pretty much any language.
Data: Data is binary and sent in form of bytes. This results in smaller payloads and efficient serialization.
Convenience: Due to the seamless mechanisms used for storing and transporting data, it is very convenient for distributed systems.
API Evolution: Contract definition using protocol buffers ensures that an API can evolve quickly using rules.

The real advantage of protocol Buffers can be seen when comparing it with the most popular mechanism of data transfer, JSON:

Smaller payload size: Since JSON needs to be stored in human-readable format, the payload is bigger than protocol buffers, where data is directly stored in binary.
For instance, comparing the same data stored in JSON vs Protocol buffers:

"person": {
    "name": 1,
    "id": 2,
    "email":3
}

The data stored in JSON amounts to 55 bytes.

message Person {
    required string name = 1;
    required int32 id = 2;
    optional string email = 3;
}

While the same data in the protocol buffer is just 20 bytes. This is a very big difference in the case of high throughput, high request rate APIs

Lesser CPU Utilisation: Since the serialization and deserialization time is negligible in protocol buffers (as data is already in binary format), it takes lesser CPU utilization to process it.
Saving Network Bandwidth: Lesser CPU Bandwidth is required due to the smaller size of the request/response packets.

Role of HTTP/2

HTTP/2 provides gRPC with some of its high performant capabilities. Some of them are:

Binary framing: Request/Response multiplexing is made possible in HTTP/2 with the introduction of a new HTTP/2 layer called binary framing.
This binary layer encapsulates and encodes the data. In this layer, the HTTP request/response gets broken down into frames.
Header compression using HPack: For cases where HTTP headers are even bigger than the payload, HTTP/2 has a very interesting strategy called HPack to handle that.
Everything in HTTP/2 is encoded before it’s sent, including the headers. HTTP/2 maps the header on both the client and the server-side. From that, HTTP/2 can know if the header contains the same value and only sends the header value if it is different from the previous header.
Multiplexing: HTTP/2 allows to send multiple requests and responses in parallel using the same TCP connection. The headers frame in Binary Framing contains typical HTTP headers information, and the data frame contains the payload. Using this mechanism, it’s possible to have data from multiple requests in a single connection.
This allows payloads from multiple requests with the same header, thus identifying it as a single request.
Server Push: Server push lets the server preemptively “push” website assets to the client without the user having explicitly asked for them. When used with care, we can send what we know the user is going to need for the page they’re requesting.
Let’s say we have a client requesting for an HTML page. We know that the client will then request for the CSS information to render the page. So we can preemptively “push” the CSS along with the HTML file.

Congratulations on making it to the end! Feel free to talk about tech or any cool projects on Twitter, Github, Medium, LinkedIn, or Instagram.

Thanks for reading!

References

More content at plainenglish.io