Building Resilient Distributed Systems

Who Is This Book For?

This book is primarily for anyone in a technical role who is helping build a distributed system. Developers, architects, testers, operations folks and SREs alike will find a lot here that will help them in their day job.

For the non-technical folks reading, there is still a lot in here for you. There is a lot of coverage of cultural and social aspects and how they can positively impact the resiliency of your distributed systems, and the second half of the book is focused on these aspects.

Whether you are struggling with achieving acceptable resiliency for an existing system, hoping to avoid making too many mistakes as you start your own journey into microservices, or looking to understand what resilience even is, then this is the book for you.

Praise for Building Resilient Distributed Systems

Resilience isn’t something you bolt onto a system. It comes from the architecture, the people, and their ability to respond when the world doesn’t behave as expected, which frankly it rarely does. Sam Newman brings those pieces together brilliantly, combining deeply practical advice with the engineering judgment behind it. As AI changes how we build and operate software, that judgment matters more than ever. This is an important book about the fundamentals we cannot afford to forget and I believe are more important than ever.

Rachel Laycock, CTO, Thoughtworks

Sam Newman has written the field guide for the day after the should-never-happen scenario happens. Clear, practical, and grounded in how systems and people actually behave, this book belongs within reach of every software architect.

Scott Hanselman, VP and member of technical staff, Microsoft/GitHub

Building resilient distributed systems is a complicated thing to do well; there are so many things to consider and so many ways that even simple things, such as one part of the system invoking behavior in another, can go wrong. Sam Newman explores them in detail in this book.

Dave Farley, coauthor of Continuous Delivery and author of Modern Software Engineering

Resiliency is one of those composite architectural characteristics that someone could write a whole book about—and Sam did! This book is a deep dive into the many facets of resiliency, covering not just the definitions but also how it intersects with other aspects of software architecture: hardware, software, and the larger organization. I wish more books like this existed, taking a comprehensive look at one critical aspect of successful modern software systems.

Neal Ford, coauthor of Fundamentals of Software Architecture, 2nd edition

Failure is a given in distributed systems. Knowing how to recover from it matters—but knowing how to prevent it in the first place, technically and organizationally, matters just as much. This book covers both, giving you the tools to build resilient systems and to handle them when they inevitably fail. Every team working on distributed systems should have this on their bookshelf.

Mark Richards, founder, DeveloperToArchitect.com, and coauthor of Fundamentals of Software Architecture, 2nd edition

An excellent guide to resilient architecture—practical, opinionated (and I agree with the opinions), and entertaining to read.

Adrian Cockcroft, OrionX.net, former Netflix and AWS architect

Wisdom-packed. Buy, read, pass it on to anyone who develops and maintains mission-critical applications.

Dr. Venkat Subramaniam, award-winning author and founder, Agile Developer, Inc.

I have been building distributed systems at large scale for 30 years (sometimes even resiliently!). There is no other single reference that lays out the hard-won techniques and trade-offs of the discipline so concisely or clearly.

Randy Shoup, SVP engineering, CircleCI

From tubthumping to thundering herds, from CAP theorem to culture and people, this book will tell you how to keep it together when your distributed systems might otherwise fall apart. Sam delivers on the promise of resilience with a catalog of patterns supported by clear explanations, practical examples, stories, and dry humor.

Kevlin Henney, coauthor of A Pattern Language for Distributed Computing

Things will go wrong in any distributed system, but the impact can be mitigated through building in resilience. I've picked up an understanding of how to do that from a variety of places over a long time—plus bitter experience. This book lays it all out for you: how to build resilient systems and respond effectively to failures.

Sarah Wells, independent tech consultant and author of Enabling Microservice Success

Resilience isn't an easy topic to write about, and yet Sam has somehow managed to jam many years of hard-won lessons into a single volume. If you're building a distributed system (and who isn't these days!), this is an essential read.

Daniel Bryant, platform engineer and PMM, Syntasso

Most resilience books stop at circuit breakers and retries. Sam goes further, showing that a truly resilient system is a sociotechnical one, where the people and processes matter as much as the architecture.

Luca Mezzalira, director, 50 Cents Media Ltd

A must-read for those interested in building large-scale systems. This book goes beyond the basic patterns than other books. It covers topics such as thundering herds, cell-based architectures, and running a service in production. It focuses on the people, process, and culture, and it’s a page-turner.

Tom Kerkove, principal software engineer, Microsoft

The book I want on the desk of every team putting agents into production. It is also the most honest writing on AI I have read this year. Sam shows names where LLMs help you build resilient systems, and where they have no business being. The teams I work with are wiring agents into everything right now. This book gives them the questions to ask first.

Benedikt Stemmildt, cofounder, hackers&wizards

Most of what I know about reliability I learned in hindsight, when it was too late. This book is all the lessons I wish I'd had upfront: practical, pragmatic, and full of ideas I'll carry into my next system.

Martha Lambert, product engineer, incident.io

Contents

The main body of the book is broken into three separate parts—Foundation, Implementation, and People. Let’s look at what each part covers.

Part I, Foundation

The first half of this book focuses on the technical aspects of making systems more resilient.

• Chapter 1: What Is Resilience?Resiliency can mean different things to different people, so it's important to start with a shared understanding. This chapter looks at resiliency from a number of angles, exploring concepts from the wider resilience engineering space, and also introducing the concept of sociotechnical systems.
• Chapter 2: Fundamentals of Observability
  Wanting your system to be resilient is one thing; knowing it’s resilient is something else. In this chapter, I’ll take you through how to collect the information you need to ensure you’re meeting your targets. I’ll also cover how to define service level objectives (SLOs), explore team versus system targets, and look at why error budgets can be useful.
• Chapter 3: Timeouts
  Networks and computers can be frustrating, and they can stop working (or go slow) at the worst time. Dealing with this fundamental truth starts with knowing when to give up - in this chapter you'll learn all about timeouts, including how to set them correctly and the importance of randomness.
• Chapter 4: Retries & Idempotency
  If at first you don't succeed, try again! Or maybe, give up? When computers or network calls start failing, trying again often makes sense, and that is the thrust of this chapter. However, trying again means we have to deal with what happens if we end up doing the same work more than once - so I'll also take you on a deep dive into the topic of idempotency.
• Chapter 5: Thundering Herds
  Systems are often brought down when there is too much demand. As I explore in this chapter, not all spikes in load are the same—these thundering herds can be internal or external, malicious or accidental, a sign of a problem or a sign of success. I’ll look at everything here from denial of service attacks to how a cache failing can bring down a system, while also presenting a range of mitigations to help fend off thundering herds of different shapes and sizes.
• Chapter 6: Rate Limiting
  This chapter explores how to reduce the amount of work your system is doing to keep it stable. It covers back pressure, load shedding, circuit breakers and more.
• Chapter 7: Resiliency and AI
  Although I cover the role of AI across the chapters in this book, I wanted one chapter to focus on the opportunities—and risks—that AI can bring. In addition to looking at the role of AI in building and operating our software, I'll also look at the impact of embedding AI in the products we build. And I'll explore the impact of AI (both positive and negative) on the human operators of our system.

Part II: Dealing with Failure

In the second part of the book, we dive into more complex technical topics and look at how to handle the failure of everything from individual computers to entire data centers. You’ll also see how a system needs to adapt when these failures occur.

• Chapter 8: Handling Machine Failure
  That computers fail isn’t a surprise, so it’s beholden on those of us building resilient software to understand the nature of these failures and work out how to mitigate them. In this chapter, you’ll see how load balancing (both client-side and server-side) can help ensure that a service remains operational even when machines fail. Additionally, I’ll explore the difference between stateless and stateful processing and explain why machine failure makes stateful processing painful to implement.
• Chapter 9: Message Brokers
  A common technique for making sure messages get delivered reliably is to make use of a message broker. In order to provide delivery guarantees, brokers themselves have to be built to handle machine failure, and this chapter will explore how this is done. Additionally, this chapter covers a number of resilience-related patterns for working with a message broker, such as the transactional outbox and consumer-managed inbox patterns.
• Chapter 10: Disaster Recovery
  One machine failing isn’t great, but having an entire data center fail is worse! In this chapter I'll discuss different patterns that can be used to recover after a large-scale outage, and I'll give you a simple model for defining your disaster recovery (DR) requirements. Also detailed here are the different options around cross-site data replication, which is essential for many of the DR patterns we explore.
• Chapter 11: Progressive Collapse
  One small issue can often cause a cascade of failures, resulting in a much more significant collapse of the wider system. This is a phenomenon known as progressive collapse. Drawing on case studies in the built environment, this chapter will look at the lessons we can learn from progressive collapse in the built environment and at how to mitigate the risk of a progressive collapse in our distributed systems.
• Chapter 12: Understanding Trade-Offs Using Formal Theorems
  When things go wrong, we often have to adapt how our system operates. There are formal theorems that can help us reason about the trade-offs available to us in our distributed system when things don’t work the way we want, and we’ll explore two of these theorems in this chapter: CAP and PACELC. This chapter will also explore Little’s Law and show how it can be useful in reducing resource contention.

Part 3: People, Process, and Culture

When considering the resiliency of a distributed system, we have to move beyond the technical and explore the behaviors, culture, and processes of the people building and maintaining the system itself.

• Chapter 13: The Sociotechnical Systems
  The concept of the sociotechnical system has been around for decades, but it is only somewhat recently that this school of thinking has come to the forefront in digital system resiliency. In this chapter, I’ll take you through the implications of approaching resiliency through a sociotechnical lens and introduce several models that can help you make sure you’re taking a holistic view of resiliency.
• Chapter 14: Incident Management and Post-Incident Response
  When things go wrong, what do you do? In this chapter, I’ll explore how to send good alerts, and how these alerts can then be handled effectively. We'll look at how to avoid operator burnout, how to coordinate incident response, and the impact of sending too many alerts.
• Chapter 15: Constant Learning and Adaptation
  To keep a system resilient, you need to continually improve. At the most basic level, this means making sure that you learn as much as possible in the wake of incidents. So in this chapter I'll take you through the importance of postmortems and give you tips on how to make sure you translate incident reports into action. Learning goes beyond this, however, so you’ll see how nurturing the right culture in your organization is vital, as well as how activities like game days can be a useful part of building a learning organization.
• Chapter 16: Conclusion
  This chapter serves as a wrap-up of the main themes and patterns covered in the preceding 15 chapters.