Distributed Systems From First Principles
Once there are two machines, everything you assumed about a single process stops being true. This is the set of trade-offs that replaces those assumptions.
10 articles · about 91 min in total
Start with Monolith vs Microservices: How to Actually DecideThe moment there are two machines, the assumptions that made single-process code easy stop holding. Calls fail halfway. Messages arrive twice. Two nodes disagree about what time it is and both are right.
Most distributed systems folklore is a compressed version of a paper that said something more careful, and the compression is where the damage happens. CAP does not mean pick two of three, it describes behavior during a partition. Exactly-once delivery does not exist, so what you actually buy is at-least-once plus an idempotent consumer.
That is why the path is ordered this way. Delivery guarantees and idempotency come early, because they are the foundation under queues, retries, webhooks and payments. Everything after them is a variation on the same theme: assume the message arrives twice and design so it does not matter.
The other recurring lesson is that the failure is rarely the component you were warned about. A circuit breaker protects you from your dependency, not your dependency from you. A health check that is too aggressive takes down a healthy fleet. Round robin distributes requests, not work.
Key takeaways
- Exactly-once message delivery does not exist in a distributed system, so the consumer must be idempotent and the guarantee you actually buy is at-least-once.
- CAP describes behavior during a network partition rather than a permanent choice of two properties, and PACELC adds the latency-versus-consistency trade made the rest of the time.
- A circuit breaker protects the caller from a failing dependency, not the dependency from the caller, which is why it does not replace rate limiting.
- An idempotency key promises that a request executes once, not that it eventually succeeds, and conflating the two produces silent data loss.
Step 1: Monolith vs Microservices: How to Actually Decide
Microservices solve an org-scale problem, not a code problem. Here is how to actually decide between a monolith and microservices, and why most startups should start with the monolith.
Jun 20, 2026 · 8 min read
Step 2: CAP Is Not “Pick Two of Three”, and It Describes the Rare Case
The famous framing is wrong twice over. Partition tolerance was never something you choose, and the trade-off only exists while a partition is happening. Which means CAP is silent about the decision you actually make every day, and PACELC is the version that names it.
Jul 29, 2026 · 9 min read
Step 3: Exactly-Once Delivery Does Not Exist. Your Consumer Has To Be Idempotent.
Not hard, not expensive, not a premium tier. Impossible, in the proven sense, and it follows from the Two Generals problem. Every product advertising exactly-once is selling at-least-once with deduplication bolted on, which means the work was always going to land on your consumer.
Jul 29, 2026 · 9 min read
Step 4: Kafka vs RabbitMQ vs SQS: Log or Queue, Not Speed
The question that actually decides your message queue is not how many messages per second you need. It is whether a message should still exist after somebody reads it. Answer that and two of the three options disappear.
Aug 28, 2026 · 10 min read
Step 5: An Idempotency Key Promises One Execution, Not Eventual Success
The design detail that surprises people: Stripe caches the result of the first request under a key even when that result was a 500, and replays the failure to every retry. That is correct, and once you see why, the rest of the design follows, including where most homegrown implementations go wrong.
Jul 29, 2026 · 9 min read
Step 6: A Circuit Breaker Protects You From Your Dependency, Not the Other Way Around
Almost every explanation frames this as being kind to a struggling downstream service. That is a side effect. Its actual job is stopping your own threads from piling up on timeouts, which is how a dependency you barely use takes down a service that does not need it.
Jul 29, 2026 · 9 min read
Step 7: Round Robin Distributes Requests, Not Work. And Your Health Check Cuts Both Ways.
Two things decide whether a load balancer helps. The algorithm, where round robin quietly assumes every request costs the same. And the health check, which is either too shallow to notice a broken server or so deep that one shared dependency marks your entire fleet unhealthy at once.
Jul 29, 2026 · 9 min read
Step 8: Rate Limiting: The Algorithm Is the Easy Part
Everyone compares token bucket to sliding window and stops there. The decisions that actually determine whether a rate limiter works are what you key it on, whether your Retry-After header is quietly scheduling a thundering herd, and whether a client that gets a 429 can safely retry at all.
Jul 29, 2026 · 9 min read
Step 9: The Three Pillars Are Not the Point. Cardinality Is, and It Is What You Get Billed For.
Logs, metrics and traces is a taxonomy of data formats, not of capability. Having all three does not mean you can answer a question you did not anticipate. What lets you do that is high-cardinality context on every request, which is also precisely what your observability vendor prices on.
Jul 29, 2026 · 9 min read
Step 10: Edge Computing in 2025: Architecting Systems That Live at the Network's Edge
A deep technical guide to building low-latency, globally distributed systems on edge runtimes, from CDN functions to stateful Durable Objects.
Apr 2, 2026 · 10 min read
Frequently asked questions
- Should I start with a monolith or microservices?
- Start with a monolith unless you already have the problem microservices solve, which is many teams blocked on one deploy. Boundaries drawn before the domain is understood are expensive to move, and a distributed system multiplies every failure mode you have not yet met.
- Is exactly-once delivery possible?
- No. If the network can drop a message and an acknowledgment can be lost, the sender cannot know whether processing happened, so it must retry. What you can build is effectively-once processing, by making the consumer idempotent so repeated delivery is harmless.
- What does the CAP theorem actually say?
- That when a network partition occurs, a distributed system must choose between staying available and staying consistent. It says nothing about normal operation, which is what PACELC adds: absent a partition, you still trade latency against consistency.
- What is an idempotency key for?
- It lets a client retry a request safely by giving the server a way to recognize it has already processed that exact operation. It guarantees the operation is applied once. It does not guarantee the operation eventually succeeds, and treating it as if it does loses data.