Distributed Services with Go

Your Guide to Reliable, Scalable, and Maintainable Systems

by Travis Jeffery

This is the book for Gophers who want to learn how to build distributed systems. You know the basics of Go and are eager to put your knowledge to work. Build distributed services that are highly available, resilient, and scalable. This book is just what you need to apply Go to real-world situations. Level up your engineering skills today.

Take your Go skills to the next level by learning how to design, develop, and deploy a distributed service. Start from the bare essentials of storage handling, work your way through networking a client and server, turn that single-node application into a distributed system with service discovery and consensus, and then deploy your service to the cloud. All this will make coding in your day job or side projects easier, faster, and more fun.

Create your own distributed services and contribute to open source projects. Build networked, secure clients and servers with gRPC. Gain insights into your systems and debug issues with observable services instrumented with metrics, logs, and traces. Operate your own Certificate Authority to authenticate internal web services with TLS. Automatically handle when nodes are added or removed to your cluster with service discovery. Coordinate distributed systems with replicated state machines powered by the Raft consensus algorithm. Lay out your applications and libraries to be modular and easy to maintain. Write CLIs to configure and run your applications. Run your distributed system locally and deploy to the cloud with Kubernetes. Test and benchmark your applications to ensure they’re correct and fast.

Dive into writing Go and join the hundreds of thousands who are using it to build software for the real world.

Q&A with Travis Jeffery:

Q: What makes this book different?

A: This book walks the reader through building a real-world software project (a self-contained, persistent event-stream service) from start to finish. There isn’t a book that walks through building a complete system like this in any language. This book introduces distributed systems practically, explaining the pieces that make up a distributed system and showing how they fit together.

Q: What will I take away after reading this book?

A: You’ll learn how distributed systems work, and how to build your own distributed services, and how to contribute to others.

Q: How is the book structured?

A: The book has four parts:

- Getting Started: Persisting events in a log;

- Network: Building a single instance networked service;

- Distribute: Distributing that service as a cluster; and

- Deploy: Deploying that cluster.

In each chapter, you learn about a problem in distributed systems (for example: consensus, service discovery, load balancing) and you build a part of your service to solve that problem. By the end, you will have built and deployed a functioning event-streaming system.

Q: How experienced with Go should I be to follow this book?

A: This book focuses on teaching how to build distributed systems using Go, and not teaching Go itself. So you should know the language. However, I walk through building the whole project code snippet by code snippet and explain what’s going on in each snippet. So if you don’t know Go, you can work your way through the book if you’re patient and use other resources to learn the syntax. I’ve also heard from people who’ve followed the book and implemented the equivalent code in other languages.

• Errata, typos, suggestions

• Source Code (zip file)

Releases:

• P1.0 2021/03/17
• B5.0 2021/03/03
• B4.0 2020/09/23
• B3.0 2020/08/12

• Introduction
• Get Started
  • Let’s Go
    • How JSON over HTTP Services Fits into Distributed Systems
    • Set Up the Project
    • Build a Commit Log Prototype
    • Build a JSON over HTTP Server
    • Run Your Server
    • Test Your API
    • What You Learned
  • Structure Data with Protocol Buffers excerpt
    • Why Use Protocol Buffers?
    • Install the Protocol Buffer Compiler
    • Define Your Domain Types as Protocol Buffers
    • Compile Protocol Buffers
    • Work with the Generated Code
    • What You Learned
  • Write a Log Package
    • The Log Is a Powerful Tool
    • How Logs Work
    • Build a Log
    • What You Learned
• Network
  • Serve Requests with gRPC excerpt
    • What Is gRPC?
    • Goals When Building a Service
    • Define a gRPC Service
    • Compile with the gRPC Plugin
    • Implement a gRPC Server
    • Register Your Server
    • Test a gRPC Server and Client
    • What You Learned
  • Secure Your Services
    • Secure Services in Three Steps
    • Authenticate the Server with TLS
    • Authenticate the Client with Mutual TLS Authentication
    • Authorize with Access Control Lists
    • What You Learned
  • Observe Your Systems
    • Three Types of Telemetry Data
    • Make Your Service Observable
    • What You Learned
• Distribute
  • Server-to-Server Service Discovery
    • Why Use Service Discovery? excerpt
    • Embed Service Discovery
    • Discover Services with Serf
    • Request Discovered Services and Replicate Logs
    • Test Discovery and the Service End-to-End
    • What You Learned
  • Coordinate Your Services with Consensus
    • What Is Raft and How Does It Work?
    • Implement Raft in Our Service
    • Multiplex to Run Multiple Services on One Port
    • What You Learned
  • Discover Servers and Load Balance from the Client
    • Three Load-Balancing Strategies
    • Load Balance on the Client in gRPC
    • Make Servers Discoverable
    • Resolve the Servers
    • Route and Balance Requests with Pickers
    • Test Discovery and Balancing End-to-End
    • What You Learned
• Deploy
  • Deploy Applications with Kubernetes Locally
    • What Is Kubernetes?
    • Install kubectl
    • Use Kind for Local Development and Continuous Integration
    • Write an Agent Command-Line Interface
    • Build Your Docker Image
    • Configure and Deploy Your Service with Helm
    • Advertise Raft on the Fully Qualified Domain Name
    • What You Learned
  • Deploy Applications with Kubernetes to the Cloud
    • Create a Google Kubernetes Engine Cluster
    • Create Custom Controllers with Metacontroller
    • Deploy to the Internet
    • What You Learned