Distributed Systems Hub

A distributed system is a set of independent computers that cooperate over a network and present themselves to users as one coherent service. Every replicated database, message broker, container orchestrator, and microservice architecture is one. This hub explains why they are hard, summarizes the few results that constrain every design, and routes you to the focused pages in this section and to the related sections on event-driven architecture, observability, and API design.

Assumed background: networking basics, concurrency, and at least one programming language. No prior distributed-systems experience is needed; the pages build from first principles.

Why Distributed Systems Are Hard

The difficulty is not incidental complexity. It follows from three physical facts that no amount of engineering removes:

1. The network is unreliable and asynchronous. Messages can be delayed, reordered, duplicated, or lost, and there is no upper bound on delay. A node that has not replied may be slow, crashed, or cut off, and waiting longer cannot tell you which.
2. Failures are partial. Some components fail while others keep running, and the survivors have an incomplete, possibly contradictory view of who is alive.
3. There is no global clock. Each node’s clock drifts independently, so wall-clock timestamps cannot reliably order events across nodes. Ordering needs explicit machinery: logical clocks, a leader, or consensus.

Some systems must also tolerate Byzantine faults, where a component behaves arbitrarily (corrupted state, buggy or malicious messages) rather than simply stopping. Most datacenter systems assume crash faults only. Byzantine tolerance appears mainly in blockchains and safety-critical avionics.

These facts are why the fallacies of distributed computing (Deutsch and others at Sun, 1994–97) still cause outages. Each is an assumption that feels true on one machine and is false across a network:

How the Pieces Fit Together

The field is a stack of consequences. Physical reality imposes theoretical limits; the coordination layer works within those limits; patterns package the coordination layer into reusable designs; technologies implement the patterns. Read top-down to learn what to build, bottom-up to learn why it has to be built that way.

flowchart TD
    Reality["Physical reality<br/>unreliable network · partial failure · no global clock"] --> Limits["Theoretical limits<br/>CAP / PACELC · FLP · Two Generals"]
    Limits --> Coord["Coordination<br/>consensus (Raft, Paxos) · replication · consistency models · failure detection"]
    Coord --> Patterns["Patterns<br/>leader election · sagas · outbox · circuit breakers · CQRS"]
    Patterns --> Tech["Technologies<br/>etcd · Kafka · Kubernetes · Istio · distributed SQL"]
    Tech --> Ops["Operations<br/>observability · SLOs · chaos and simulation testing"]
    Ops -. "feedback: what actually fails" .-> Patterns

Results Every Design Must Respect

CAP and PACELC

CAP (Brewer’s conjecture, proved by Gilbert and Lynch in 2002): while a network partition is in progress, a replicated system can offer linearizable consistency or availability (every request to a live node gets a non-error answer), but not both. Partitions cannot be ruled out in a real network, so the practical choice is what to give up during a partition:

• CP: refuse or block requests on the minority side to stay consistent. Examples: etcd, ZooKeeper, Consul’s catalog, Spanner, CockroachDB.
• AP: keep answering on both sides and reconcile afterwards. Examples: Cassandra, Riak, DynamoDB with its default eventually consistent reads.

CAP says nothing about the normal case. PACELC (Abadi, 2012) completes it: if partitioned, choose Availability or Consistency; else, choose Latency or Consistency. The “else” half is the trade-off you pay on every request, because strong consistency requires cross-replica coordination even when the network is healthy.

flowchart TD
    Q{"Is the network<br/>partitioned?"} -- yes --> P{"Give up..."}
    P -- "availability" --> PC["PC: minority side rejects requests<br/>etcd, ZooKeeper, Spanner"]
    P -- "consistency" --> PA["PA: all sides answer, reconcile later<br/>Cassandra, Riak"]
    Q -- no --> E{"Trade off..."}
    E -- "latency" --> EC["EC: coordinate on every write<br/>(quorum or leader round trip)"]
    E -- "consistency" --> EL["EL: answer from the nearest replica<br/>(may be stale)"]

FLP Impossibility

Fischer, Lynch, and Paterson (1985) proved that no deterministic protocol can guarantee consensus in a fully asynchronous system if even one process may crash. Practical protocols such as Raft and Paxos therefore always preserve safety (never decide two different values) but guarantee liveness (eventually decide) only when the network behaves for long enough. They get there with timeouts, randomized election delays, or partial-synchrony assumptions.

Formal statements and proofs are in Distributed Systems Theory. The engineering consequences are in Consensus & Coordination.

Consistency Is a Dial

Stronger guarantees cost more coordination, and so more latency and less availability. Choose the weakest model the application can tolerate. From strongest to weakest:

Session guarantees (read-your-writes, monotonic reads, monotonic writes, writes-follow-reads) make weak models tolerable for a single user without paying for global coordination. Transactional isolation (serializable, snapshot isolation) is a separate axis about multi-object operations. Strict serializability combines it with linearizability.

Topics in This Section

The pages are ordered so that concepts come before patterns. Start with the limits that constrain every design, then move to the patterns and infrastructure that work within them.

Concepts and Foundations

Patterns and Infrastructure

Suggested Reading Order

flowchart LR
    C["Consensus and<br/>Coordination"] --> R["Replication"]
    R --> F["Failure Detection"]
    F --> CS["Client-Side<br/>Consistency"]
    C --> M["Microservices and<br/>Event-Driven"]
    M --> RP["Resilience"]
    M --> SD["Service Discovery"]
    RP --> O["Observability"]
    SD --> O
    O --> T["Testing and Chaos"]

The left branch explains how data stays correct across replicas; the right branch explains how services are composed and kept running. Both depend on the limits in Consensus & Coordination.

Related Sections

Several neighbouring sections go deeper on topics this one introduces:

Design Principles

• Design for failure. Assume every node, link, and dependency will fail. Timeouts, retries with backoff, idempotency, and circuit breakers turn failure from an outage into routine noise.
• Choose consistency per operation. Decide deliberately where you need linearizability (uniqueness, money, leadership) and where eventual consistency is fine. Document the choice.
• Keep services stateless where possible. Put state in purpose-built stores so compute scales horizontally and recovers by restarting.
• Make every side effect idempotent. At-least-once delivery is the norm, so duplicates must be harmless.
• Observe from the start. Propagate trace context across every boundary and define SLOs before the first incident, not after.
• Start simple. A modular monolith with clear internal boundaries is cheaper to run and easier to split later than a premature microservice mesh.

See Also

• Kubernetes: container orchestration and cluster management
• Docker: containerization fundamentals
• AWS Cloud Services: managed distributed infrastructure
• Database Design: sharding, replication, and consistency in data stores
• Networking: the unreliable substrate every distributed system runs on
• CI/CD Pipelines: progressive delivery and rollouts for distributed services

Further Reading

• Martin Kleppmann and Chris Riccomini, Designing Data-Intensive Applications, 2nd ed. (O’Reilly, 2026)
• Maarten van Steen and Andrew S. Tanenbaum, Distributed Systems, 4th ed. (free PDF at distributed-systems.net)
• Betsy Beyer et al., Site Reliability Engineering and The Site Reliability Workbook (Google, free online at sre.google/books)
• DeCandia et al., Dynamo: Amazon’s Highly Available Key-value Store (SOSP 2007)
• Ongaro and Ousterhout, In Search of an Understandable Consensus Algorithm (Raft, USENIX ATC 2014)
• MIT 6.5840: Distributed Systems (formerly 6.824), lectures and labs