Building Resilient Distributed Systems

How To Read The Book?

The book is in the final editing stages, but if you can't wait, you can head over to O'Reilly's website to read two thirds of the book that are available in early access form. If you sign up for an O'Reilly subscription with the discount code SAM20 you'll get 20% off a subscription (and I earn a little bit too).

The book is available to pre-order from bookshops now. Whilst you can of course order it from Amazon, I'd prefer it if you supported your local independent bookshop and order it there instead if you can. For those in the UK, you can pre-order the book at Hive, who actively support small local booksellers. ,

Who Is This Book For?

This book is primarily for anyone in a technical role who is helping build a distributed system. Developers, architects, testers, operations folks and SREs alike will find a lot here that will help them in their day job.

For the non-technical folks reading, there is still a lot in here for you. There is a lot of coverage of cultural and social aspects and how they can positively impact the resiliency of your distributed systems, and the second half of the book is focused on these aspects.

Whether you are struggling with achieving acceptable resiliency for an existing system, hoping to avoid making too many mistakes as you start your own journey into microservices, or looking to understand what resilience even is, then this is the book for you.

Feedback

If you have any feedback after reading the early access, then please contact me and let me know!

Planned Contents

This book is still being written. As such, the table of contents is subject to change. If there is something missing here that I should include, please feel free to send me some feedback

The main body of the book is broken into three separate parts—Foundation, Implementation, and People. Let’s look at what each part covers.

Part I, Foundation

The first half of this book focuses on the technical aspects of making systems more resilient.

• Chapter 1: What Is Resilience?Resiliency can mean different things to different people, so it's important to start with a shared understanding. This chapter looks at resiliency from a number of angles, exploring concepts from the wider resilience engineering space, and also introducing the concept of sociotechnical systems.
• Chapter 2: Fundamentals of Observability
  Wanting your system to be resilient is one thing; knowing it’s resilient is something else. In this chapter, I’ll take you through how to collect the information you need to ensure you’re meeting your targets. I’ll also cover how to define service level objectives (SLOs), explore team versus system targets, and look at why error budgets can be useful.
• Chapter 3: Timeouts
  Networks and computers can be frustrating, and they can stop working (or go slow) at the worst time. Dealing with this fundamental truth starts with knowing when to give up - in this chapter you'll learn all about timeouts, including how to set them correctly and the importance of randomness.
• Chapter 4: Retries & Idempotency
  If at first you don't succeed, try again! Or maybe, give up? When computers or network calls start failing, trying again often makes sense, and that is the thrust of this chapter. However, trying again means we have to deal with what happens if we end up doing the same work more than once - so I'll also take you on a deep dive into the topic of idempotency.
• Chapter 5: Thundering Herds
  Systems are often brought down when there is too much demand. As I explore in this chapter, not all spikes in load are the same—these thundering herds can be internal or external, malicious or accidental, a sign of a problem or a sign of success. I’ll look at everything here from denial of service attacks to how a cache failing can bring down a system, while also presenting a range of mitigations to help fend off thundering herds of different shapes and sizes.
• Chapter 6: Rate Limiting
  This chapter explores how to reduce the amount of work your system is doing to keep it stable. It covers back pressure, load shedding, circuit breakers and more.
• Chapter 7: Resiliency and AI
  Although I cover the role of AI across the chapters in this book, I wanted one chapter to focus on the opportunities—and risks—that AI can bring. In addition to looking at the role of AI in building and operating our software, I'll also look at the impact of embedding AI in the products we build. And I'll explore the impact of AI (both positive and negative) on the human operators of our system.

Part II: Dealing with Failure

In the second part of the book, we dive into more complex technical topics and look at how to handle the failure of everything from individual computers to entire data centers. You’ll also see how a system needs to adapt when these failures occur.

• Chapter 8: Handling Machine Failure
  That computers fail isn’t a surprise, so it’s beholden on those of us building resilient software to understand the nature of these failures and work out how to mitigate them. In this chapter, you’ll see how load balancing (both client-side and server-side) can help ensure that a service remains operational even when machines fail. Additionally, I’ll explore the difference between stateless and stateful processing and explain why machine failure makes stateful processing painful to implement.
• Chapter 9: Message Brokers
  A common technique for making sure messages get delivered reliably is to make use of a message broker. In order to provide delivery guarantees, brokers themselves have to be built to handle machine failure, and this chapter will explore how this is done. Additionally, this chapter covers a number of resilience-related patterns for working with a message broker, such as the transactional outbox and consumer-managed inbox patterns.
• Chapter 10: Disaster Recovery
  One machine failing isn’t great, but having an entire data center fail is worse! In this chapter I'll discuss different patterns that can be used to recover after a large-scale outage, and I'll give you a simple model for defining your disaster recovery (DR) requirements. Also detailed here are the different options around cross-site data replication, which is essential for many of the DR patterns we explore.
• Chapter 11: Progressive Collapse
  One small issue can often cause a cascade of failures, resulting in a much more significant collapse of the wider system. This is a phenomenon known as progressive collapse. Drawing on case studies in the built environment, this chapter will look at the lessons we can learn from progressive collapse in the built environment and at how to mitigate the risk of a progressive collapse in our distributed systems.
• Chapter 12: Understanding Trade-Offs Using Formal Theorems
  When things go wrong, we often have to adapt how our system operates. There are formal theorems that can help us reason about the trade-offs available to us in our distributed system when things don’t work the way we want, and we’ll explore two of these theorems in this chapter: CAP and PACELC. This chapter will also explore Little’s Law and show how it can be useful in reducing resource contention.

Part 3: People, Process, and Culture

When considering the resiliency of a distributed system, we have to move beyond the technical and explore the behaviors, culture, and processes of the people building and maintaining the system itself.

• Chapter 13: The Sociotechnical Systems
  The concept of the sociotechnical system has been around for decades, but it is only somewhat recently that this school of thinking has come to the forefront in digital system resiliency. In this chapter, I’ll take you through the implications of approaching resiliency through a sociotechnical lens and introduce several models that can help you make sure you’re taking a holistic view of resiliency.
• Chapter 14: Incident Management and Post-Incident Response
  When things go wrong, what do you do? In this chapter, I’ll explore how to send good alerts, and how these alerts can then be handled effectively. We'll look at how to avoid operator burnout, how to coordinate incident response, and the impact of sending too many alerts.
• Chapter 15: Constant Learning and Adaptation
  To keep a system resilient, you need to continually improve. At the most basic level, this means making sure that you learn as much as possible in the wake of incidents. So in this chapter I'll take you through the importance of postmortems and give you tips on how to make sure you translate incident reports into action. Learning goes beyond this, however, so you’ll see how nurturing the right culture in your organization is vital, as well as how activities like game days can be a useful part of building a learning organization.
• Chapter 16: Conclusion
  This chapter serves as a wrap-up of the main themes and patterns covered in the preceding 15 chapters.