Scaling from 100 to 100 Million Users

Scroll sideways to see the whole diagram.

The first time I saw a diagram like that, my honest reaction was to close the tab. It looks like a hundred decisions made at once, and I assumed you had to design all of it up front. It turns out you don't, and that was the first thing that made the whole topic less scary: every box was added on a specific day, because a specific number on a specific dashboard turned red, and each one starts exactly where the previous one broke.

So this is me walking that path to try to understand it, with one imaginary product, Reelo, a short-video sharing app, from 100 users on a single server to 100 million across regions. At every rung I kept asking the same three questions: which part is actually struggling, what is the smallest fix, and what does that fix cost, in latency, consistency, complexity and real dollars.

The thing I didn't expect to spend so much time on is when not to scale. Distributing a system before it needs to be distributed seems to be the single most expensive mistake in this whole story, and it's the one I'd have walked straight into.

Here's the ladder we'll climb:

Basics: three words, two directions, four axes

Latency, throughput, capacity

These three blurred together for me constantly at first, and most of my confused reading cleared up once I kept them apart, so let me pin them down first.

• Latency is how long one request takes.
• Throughput is how many requests you complete per second.
• Capacity is how much load you can absorb before things start to degrade.

They move independently, which is exactly why it helps to keep them apart. Google's SRE book builds its monitoring around four "golden signals" (latency, traffic, errors and saturation), and they map cleanly onto these. The catch that took me a while to notice: buying throughput or capacity usually costs some latency in return. Another hop, a queue, a coordination step. I've started thinking of scaling as a dial you turn, not a switch you flip.

Up or out

There are only two directions. Make the one box you have bigger, or put more boxes next to it.

Vertical is where you start. It is boring, and it costs zero architectural complexity and zero code changes. I'll come back to exactly where it stops working, because that surprised me too.

Which of four things are you actually scaling?

As far as I can tell, every scaling problem comes down to pressure on one of four things. Figuring out which one turned out to be most of the work.

Once you know which one is under pressure, there are two ways to split it. You can split by what the system does (an upload service, a feed service, a notification service), or by which data it holds (one key range on one shard, the next range on another).

Split by function first, because it is reversible. Splitting by data is close to permanent, so save it for later.

100 users: one box, and that's correct

Reelo starts life the way most products should: a monolith on one server, one PostgreSQL database, and Nginx in front.

This is not a stage to rush past. It is fast, it is easy to debug, and it is small enough that one person can hold the whole thing in their head. Every network hop you add later is a hop that can be slow, fail, or disagree with its neighbor.

The most expensive mistake I kept reading about is distributing a system before it needs to be. So we don't. Not yet.

But one thing does have to exist before anything breaks: the ability to see it coming.

Instrument while it's calm

Two standard methods cover a system like this, and you want both. Tom Wilkie's RED method watches your endpoints: rate, errors and duration of requests. Brendan Gregg's USE method watches your resources: utilization, saturation and errors for each CPU, disk, NIC and pool. RED tells you users are hurting; USE tells you which piece of hardware is the reason.

Averages lie

The thing that surprised me most here: two systems can report the same average latency while one of them leaves a slice of real users waiting on a slow tail the average hides. Once I saw that, I understood why people track percentiles instead. The p50 is what a typical request feels like. The p99 is what your unluckiest request in a hundred feels like, and it's usually the one somebody complains about.

One number I wouldn't have thought to track on day one, while nothing is on fire: cost per thousand requests. It's the metric that ends up keeping the later stages honest.

Dashboards say "slow." Traces say where.

A dashboard tells you something is slow. It won't tell you which part. Tag a request with an ID at the edge, carry that ID through every hop, and what comes back is a trace: one bar per service, laid end to end.

Find your ceiling on purpose

With instruments in place, use the calm to find your limit deliberately. Push the system with a load test until something gives, and watch which numbers move first. Finding the ceiling in a load test costs an afternoon. Finding it in production costs an outage.

10,000 users: one bar turns red

Axis: compute. Then the traffic arrives: a hundred times what you had yesterday. One bar on the dashboard turns red, and it's the CPU. The request handlers themselves are the bottleneck.

The fix is the simplest one in this entire post: move to a bigger instance. No code changes, just a resize and a restart. And the pricing holds a pleasant surprise. Across a general-purpose family like AWS's M6i, from the smallest size to the largest, on-demand cost works out to the same rate per vCPU. Double the instance, double the cost, double the capacity. There is no premium at the top; the price step people actually feel is moving off shared, burstable instances onto dedicated ones.

Scaling up isn't expensive. It just has a ceiling.

Read the query plan before you buy hardware

Here's what I didn't expect: often the hardware isn't the bug at all. Datadog wrote up a case where changing a single line of SQL let Postgres use an index that already existed: a query that took about 22 seconds dropped to roughly 200 milliseconds, around 110 times faster, with no hardware touched. Sentry's blog walks through a classic N+1, an endpoint firing 150-odd sequential queries in a loop, collapsed into one join. The p95 fell from about 7 seconds to under 2, and the median from 3 seconds to 275 milliseconds.

Hardware gives you a dependable 2× for twice the money. A query fix can give you two orders of magnitude for free.

A missing index has beaten a server upgrade many times over.

Where vertical scaling actually stops

Two places, and neither one is the price tag.

1. The ceiling. Every instance family has a largest size, and eventually your load passes it. There is simply no bigger box to buy.
2. The blast radius. One box is one failure domain. One bad deploy, one kernel patch, one maintenance window, and the entire product is down with it.

More boxes work, right up until they don't

So you add boxes. It's worth knowing the limit of that move before you build it. Add concurrency to any system and throughput rises the way you'd expect, then flattens, and past a certain point it actually falls. Neil Gunther published the math for this in 1993 as the Universal Scalability Law. Two forces fight the linear ideal: contention (work waiting on a shared resource, like a lock or a single primary) and coherency (the cost of keeping nodes in agreement, which grows with every pair of nodes). It isn't only a multi-threading curiosity, either: Gunther and colleagues later fit the same curve to a real multi-node Hadoop cluster.

Scale the bottleneck, not the system. Adding more of everything is not the same as adding more of the one thing that's actually constrained.

100,000 users: no bigger box left to buy

Axis: compute, horizontally.

Empty the box before you copy it

A second server only helps once the first one stops remembering things. Right now it's holding state it shouldn't: sessions in memory, uploaded files on local disk, a cache inside the process, a cron job living inside the app. Each one has to leave before cloning the box is safe.

Scroll sideways to see the whole diagram.

There's a trap here that works well enough to be dangerous: sticky sessions. Pinning each client to one box for the life of its session lets you skip the hard part, and it mostly works. But it spreads load unevenly, it makes every deploy and every instance failure log people out or lose their work, and the Twelve-Factor App calls it out as a violation outright. Move the state; don't pin the user.

How traffic finds the right box

A request passes through three layers, and each does a genuinely different job.

Scroll sideways to see the whole diagram.

DNS steers you toward a region. Layer 4 makes a fast, raw decision about which box gets this connection without looking inside it. Layer 7 actually reads the request (path, headers, cookies) and routes on that, which is also why request throttling lives there: you can't rate-limit by API key without reading the key.

One assumption I had to drop: TLS is not the dividing line. AWS's Network Load Balancer, a layer 4 product, terminates TLS. The real difference is content-based routing, which only layer 7 can do.

The health check that causes the outage

Health checks come in two depths. A shallow check asks only "is the process alive?" A deep check reaches further: auth, a queue, the database, every dependency the service relies on. That sounds safer, and it's exactly what makes it dangerous.

One shared blip just became a total, self-inflicted outage. The fix: keep liveness checks shallow, and be deliberate about what a readiness check depends on. A dependency that every instance shares tells you nothing about any single instance. (It's also why some load balancers, AWS's included, "fail open" and send traffic to every target when all of them look unhealthy at once.)

A health check can be the outage.

Autoscale on the signal that's actually saturated

The fleet survives health checks now. Does it grow fast enough for a real spike? Only if it's watching the right number. AWS documents the classic failure: a fleet waiting on a slow dependency has every thread occupied while CPU sits low, because the workers are waiting, not computing. A CPU-based autoscaler sees headroom that doesn't exist and does nothing. The fix is to scale on a number that reflects the real bottleneck, such as backlog per instance or in-flight requests.

Even the right signal isn't instant. A new instance needs time to boot, start the app and warm its caches before it's useful. When Pokémon GO launched, Google Cloud reported that traffic surged past Niantic's expectations within 15 minutes of going live in Australia and New Zealand, eventually reaching about 50 times the launch target and 10 times the worst-case estimate. No scaling policy is tuned for 10× your worst case. Getting through it took engineers from both companies working the problem live.

Autoscaling saves you money. It doesn't save you from a spike. For launches you know about, scale ahead of time.

1 million users: everyone asks the database the same question

Axis: reads. Autoscaling gave you fifteen app servers. All fifteen are now asking one database for data, and the database is the new red bar.

Pagination: stop paying for rows you skip

Showing a long list a screenful at a time is pagination, and there are two ways to do it that look identical to the user and behave nothing alike under load. Naming them is the first step, because the slow one is the default almost every tutorial teaches.

The obvious approach is offset pagination: "skip the first 10,000 rows, give me the next 20," written LIMIT 20 OFFSET 10000. The trouble is hiding in the word skip. A database has no way to teleport to the 10,000th row. To honor OFFSET 10000 it walks the first 10,000 rows in sorted order, reads each one, and throws it away, and only then starts collecting the 20 you actually asked for. Page one is instant. Page five hundred reads a hundred thousand rows to return twenty of them. The cost of a page grows linearly with how deep into the list it sits.

Scroll sideways to see the whole diagram.

Keyset pagination (also called cursor or seek pagination) fixes this by changing the question. Instead of "skip N rows," it asks "give me the rows after the last one I already saw." You stop remembering a position and start remembering a value, the sort key of the last row on the current page, and ask for everything past it:

-- OFFSET: the database walks and discards 10,000 rows first.
-- Cost grows with depth. Page 500 is 500x the work of page 1.
SELECT * FROM clips ORDER BY id DESC LIMIT 20 OFFSET 10000;

-- KEYSET: :after_id is the id of the last row the client has.
-- The index seeks straight to it and reads exactly 20 rows.
-- Every page costs the same, whether it's page 1 or page 10,000.
SELECT * FROM clips WHERE id < :after_id ORDER BY id DESC LIMIT 20;

Why does the rewrite solve the problem rather than just move it? Because id is indexed and the list is already ordered by it, so the database can seek directly to the one entry in the index equal to :after_id and read the next twenty in order. It never touches the rows before it. You've replaced a scan-and-discard whose cost scales with depth with a single index seek whose cost is constant.

Read replicas, and the lag you can't remove

Even a cheap query is still one database answering fifteen servers. The first fix that comes to mind is read replicas. A replica is a full, independent copy of the database that stays continuously in sync with the original. You designate the original as the primary (it alone accepts writes) and stand up several replicas that accept only reads. Your app then sends every INSERT/UPDATE/DELETE to the primary and spreads its SELECTs across the replicas.

Why does that scale reads? Because reads were the bottleneck, and you've gone from one machine answering all of them to N machines answering a share each. Three replicas means roughly a third of the read load lands on any one of them. Writes don't scale this way, note, every write still has to be applied everywhere, which is exactly why this chapter is about reads and the next one is about writes.

GitHub, which runs a large MySQL fleet, has written that it commonly expects sub-second replication lag and pulls lagging replicas out of service within seconds. That is about as good as this gets. Well under a second is still not zero.

Scroll sideways to see the whole diagram.

That not-quite-zero lag has a user-visible consequence. A user uploads a clip (write → primary), gets redirected to their profile, and that read lands on a replica that is 300 ms behind and hasn't received the new row yet. Their own upload is invisible to them for a moment, so they shrug and upload it again. The guarantee they expected has a name, from Werner Vogels's 2008 essay on eventual consistency: read-your-writes, the promise that you will always see the effect of your own writes, even if other people's take a moment to show up.

The fix is not to eliminate the lag (you can't, cheaply). It's to make the guarantee selective, so only the person who just wrote pays for it:

• Read-from-primary window. For a short period after a user writes (say, a few seconds, longer than your typical lag), route that user's reads to the primary. Everyone else keeps reading replicas. Simple, and it covers the redirect-after-upload case exactly.
• Replication-position tracking. On a write, capture the primary's log position (the LSN in Postgres). Attach it to the user's session, and on their next read pick a replica that has already replayed up to at least that position. More precise, more plumbing.

Every fix trades away a guarantee. Here you trade global consistency, replicas are allowed to be stale, for a cheaper per-user consistency that's all the user actually notices.

Cache-aside: the highest-leverage lever in this post

Reads are off the primary now. Are they fast? A cache is a small, fast store (an in-memory key-value service like Redis or Memcached) that holds the answers you look up most often, so you can skip the slow database lookup entirely. The question is how the cache and the database stay coordinated, and the most common answer is a pattern called cache-aside (or "lazy loading"), where the application, not the database, owns that coordination:

1. Read: check the cache for the key. On a hit, return it, done, the database never hears about this request. On a miss, read from the database, write the result into the cache with an expiry, then return it. The next reader of that key gets a hit.
2. Write: update the database, then invalidate (delete) the cached key so the next read reloads fresh data.

It's called cache-aside because the cache sits off to the side of the database rather than in front of it; the app talks to both and decides what goes where.

Why is this the highest-leverage move in the whole post? Because the saving is governed by one number, the hit rate, and it's non-linear. If 90% of reads are hits, the database sees 10% of the traffic, a 10× reduction. Push the hit rate to 99% and the database sees 1%, a 100× reduction. The last few percent of hit rate are worth more than everything before them.

Facebook's paper on scaling memcache tracked production pools over a week and reported hit rates from the low 90s up past 99.9 percent. Even at the low end, the database sees under a tenth of the read traffic it would otherwise take. Nothing else in this post buys that much for so little.

When the cache goes stale

The moment you keep a second copy of data, you own a consistency problem: the database (the source of truth) can change while the cache still holds the old value. A cached value that no longer matches the source is stale, and the whole art of caching is deciding how you let staleness happen and for how long. There are three standard answers, and each has a failure mode worth knowing.

Most real systems use TTL as a safety net under explicit delete: delete keys on write for freshness, but also give them a TTL so any key a buggy path forgot to invalidate still self-heals within the window.

Cache the answer, not the row

All three strategies above assume you're caching a row, a single record keyed by its id. The home feed isn't a row. Building it joins several tables (who you follow, their recent clips, engagement counts), sorts the result by time or rank, and cuts it to a screenful. No single key holds that answer, and recomputing it on every page load is exactly the expensive work you're trying to avoid.

So cache the answer instead of its ingredients: precompute the finished feed and store it under one key, ready to read back in a single hit. A stored, precomputed query result like this is a materialized view (as opposed to a normal view, which re-runs the query every time it's read). You've moved the work from read time to write time, and that's the trade you're choosing deliberately: the same fact now lives in two places, so every write must update both, and the view is only as fresh as its last rebuild.

So who rebuilds it, and when? This is the famous feed-design fork, and it has two answers.

Scroll sideways to see the whole diagram.

• Fan-out on write (push). The moment someone posts a clip, push its id into the precomputed feed of every one of their followers. Reads are then trivially cheap, the feed is already sitting there assembled, but a single post costs as many writes as the author has followers.
• Fan-out on read (pull). Write nothing extra on post. When a user opens the app, then gather the recent clips of everyone they follow, merge, sort, and return. Writes are cheap; every read pays the full assembly cost.

A feed is read far more often than it's written, so doing the expensive work once at write time, and cheaply on every read, usually wins. That holds for almost every account, and then there's the one it doesn't. Someone with ten million followers turns a single upload into ten million feed inserts, a write storm that can stall the whole system, and you've rebuilt the hot-partition problem on the write side. So real systems (Twitter's timeline being the canonical example) do both: fan out on write for normal accounts, keep the handful of celebrity accounts out of the push path, and merge their clips in at read time. Each reader's feed is "precomputed feed, plus a quick pull of the few huge accounts I follow."

The thundering herd

Here's how the cache that was saving you turns around and knocks over the database behind it. Picture one very popular key, the feed of a creator a hundred thousand people are watching, served from cache thousands of times a second. Its TTL expires. For one instant the key is simply gone, and every single one of those concurrent requests misses at the same moment. Cache-aside says "on a miss, read the database and refill," so now all of them do, at once, because none has finished refilling yet to save the others. The database goes from near-zero load on that key to thousands of identical queries in the same breath, and that stampede on its own can take it down. This is the thundering herd (or cache stampede): not a slow key, but a synchronized swarm of misses all doing the same expensive reload.

The fix is to notice that all those requests want the same answer, so only one of them should actually go compute it. Facebook measured this directly in the same memcache paper; their mechanism is called a lease. On a miss, the cache hands exactly one request a short-lived token granting it the right to refill the key. Every other request for that key is told "someone's already fetching it, wait a moment," and they poll the cache instead of piling onto the database. The one holder reads the database once, writes the value back, and everyone else immediately gets a hit.

That's peak database load from Facebook's memcache paper (NSDI 2013): a 92 percent cut, measured rather than estimated.

Jittering the expiry times attacks a different flavor of the same problem. When a batch of keys is loaded together, after a deploy, a cache flush, or a warm-up job, they all get the same TTL and therefore all expire on the same tick, producing a herd even without a single hot key. The fix is to add a small random offset to each TTL (say 600s ± up to 60s) so keys born together die spread out. One line of code, and a synchronized cliff becomes a gentle trickle of misses the cache can refill without noticing.

The key that's simply too popular

A cache cluster is more than one machine, because no single box has the memory or the network card to hold and serve everything. So the keyspace is split across shards, and each key is assigned to one of them, usually by hashing the key and taking it modulo the number of shards (shard = hash(key) % N). Spread evenly, every shard carries a fair slice of the load. That's the assumption.

A single hot key breaks it. One viral clip means one key, and that key hashes to exactly one shard. Every request for the viral clip, which may be a large fraction of all your traffic, lands on that one shard while the rest of the cluster sits idle. The problem isn't total capacity, it's skew: the load is real but it's all pointed at one machine. And this is why the instinctive fix fails, adding more shards changes nothing, because the hot key still hashes to a single shard; you've just given the idle machines company. Redis documents this by name and ships a flag to help you find such keys.

Twitter hit a sharp version in April 2018: during a planned datacenter failover test, a hash function that ignored the last byte of keys concentrated a set of large, hot values onto a handful of cache shards. Those shards' network cards saturated, and the site was partially down for about an hour, textbook skew, not a capacity shortage.

Why does a local cache fix skew when more shards don't? Because it attacks the problem at the reader, not the storage. Each app server keeps the hot value in its own memory for even a second or two; during that window its thousands of requests for the key are answered locally and collapse into a single read against the shared shard per server. Fifteen servers hammering one shard become fifteen reads a second. The cost is a little extra staleness (up to that local TTL), which for a view count or a trending clip nobody minds.

And here's a harder question than the hot key itself: if the entire cache disappeared right now, would your system survive in some degraded form, or fall over? If the honest answer is "fall over," the cache isn't an optimization anymore.

A cache your database can't survive without is an undocumented dependency. Treat it like one.

The one limit no upgrade fixes

Light in optical fiber travels at roughly two thirds of its speed in a vacuum, about 200,000 km per second. Sydney to Virginia is about 15,700 km, which puts a hard floor of about 157 ms on the round trip before a single router, server or line of code touches the request.

That's a physical floor on latency, and no amount of compute changes it. It's where a CDN's value comes from: serving a request from a point near the user is a distance problem, not a compute problem.

No server upgrade beats the speed of light.

5 million users: writes start piling up

Axis: writes. Reads are handled end to end now. Uploads are what's backing up. This is the hard axis: you can copy data to scale reads, but every write eventually has to land somewhere authoritative.

Connections run out before queries do

A database connection is not free bookkeeping. In PostgreSQL, every open connection is backed by its own operating-system process on the database server, with its own memory and scheduling cost, whether or not it's currently running a query. That changes the arithmetic. Fifteen app servers, each keeping a local pool of 100 connections, means 1,500 processes converging on one database, which ships with max_connections set to 100 for exactly this reason. AWS measured idle connections holding on the order of a megabyte and a half of memory each, and a couple of thousand idle ones measurably raising CPU on a small instance. A database can run short of resources purely from holding connections open, before it answers a single query.

Scroll sideways to see the whole diagram.

The fix is a connection pooler (PgBouncer, or RDS Proxy on AWS): a small proxy that sits between the fleet and the database. The fleet opens its 1,500 cheap connections to the pooler, and the pooler keeps a small, fixed set of real connections to the database, handing them out to whichever app connection actually needs to run a query right now. The database's process count stays flat no matter how large the fleet in front of it grows, this is multiplexing, many logical connections sharing few physical ones.

One caution, because it bites people: poolers run in different modes. Session pooling pins a real connection to a client for its whole session (safe, but barely better than no pooler). Transaction pooling, the mode that gives the big win, hands the real connection to a different client after every transaction, so anything that relies on connection-level state across transactions, temporary tables, session settings, advisory locks, and on older PgBouncer versions prepared statements, can break. Check the mode against what your ORM assumes before you flip it on.

Batch the writes, budget the indexes

Connections are under control, but two hidden costs still make writes slower than they need to be.

The first is per-statement overhead. Every individual INSERT carries a network round trip to the database and a commit (a durable flush to disk) of its own. Send 1,000 rows as 1,000 separate inserts and you pay that overhead a thousand times. Batching, sending many rows in one multi-row INSERT inside one transaction, amortizes it: one round trip, one commit, for the whole batch. The rows written are identical; you've just stopped paying the postage on every envelope separately.

The second is subtler and permanent: indexes. An index is a secondary sorted structure the database maintains so reads can find rows fast (it's what made keyset pagination and replica reads cheap). But it's a copy of part of your data kept in sorted order, so an INSERT doesn't just append a row to the table, it also has to insert that row's key into every index, each in the right sorted position. Three indexes on a table means every insert does one table write plus three index writes.

Your read optimization is your write problem. Every index you added in the reads chapter is now a tax on every upload. Keep the ones that earn their keep; delete the rest.

Split by function first, then by key

One database can only take so much of anything. There are two fundamentally different ways to cut it up, and the order you try them in matters.

Functional split (split by capability) comes first, because it's the reversible one. You move whole feature areas onto their own databases: uploads get one, the feed gets another, notifications get a third. Each database now holds fewer tables and takes only its feature's share of the load. It's reversible because the tables themselves are untouched, if it doesn't help, you can move them back. But it has a real cost, paid immediately: a query that used to JOIN across uploads and feed in one place can't anymore (they're different databases), and an operation that used to be one atomic transaction across those tables becomes several separate ones that can partially fail.

Scroll sideways to see the whole diagram.

When even one of those databases outgrows a single machine, you shard (also called horizontal partitioning): split one table by which data each row holds, not by what feature it belongs to. You pick a shard key (say, the clip's id or the creator's id) and a rule that maps each key to one shard. Picture a bookshelf divided by letter, A–F on one shelf, G–L on the next; every clip is filed on the shelf matching its key. Done well, each shard holds a fraction of the rows and takes a fraction of the traffic, so you've scaled writes (and storage) the way replicas scaled reads, evenly, at first.

Then reality arrives. Each partition has its own ceiling no matter how much capacity the table has overall; a single DynamoDB partition, for example, tops out at 3,000 reads and 1,000 writes per second. One popular creator can overload a single shelf while the other three sit comfortably idle. Real traffic drifts toward a subset of keys over time, which is why DynamoDB now applies adaptive capacity to every table by default.

Changing the number of shards

Here's the permanent part, and the reason sharding is a decision you don't walk back. One shelf is overloaded, so the number of shards has to grow. The naive mapping rule, shard = hash(key) % N, has a catastrophic property: change N and you change the answer for almost every key, not just the ones that need to move. Adding a single shard triggers a near-total data migration while the system is live.

Consistent hashing fixes this by changing the mapping so that adding a shard disturbs only a small, local slice of keys. Imagine the hash space bent into a ring (0 at the top, wrapping back around). You place each node at a point on the ring, and each key belongs to the first node you reach going clockwise from the key's position. Add a new node and it inserts itself at one point on the ring; the only keys that move are those sitting between the new node and its counter-clockwise neighbor, everything else stays exactly where it was. A node leaving is the mirror image: only its keys spill to the next node clockwise. It's the technique at the heart of Amazon's 2007 Dynamo paper, and it's why Cassandra gives every node many virtual positions on the ring (so the load it sheds or absorbs is spread across many neighbors instead of dumped on one).

Scroll sideways to see the whole diagram.

There's a cost beyond the migration itself. Queries that cross shards get slower and harder: a simple count has to visit every shard and sum the answers, and a transaction spanning shards becomes a distributed transaction, a genuinely harder problem. Choose the shard key around your most important access pattern, and keep the operations that must be atomic inside one shard.

Resharding seems to be the operation everyone dreads, unless they planned for it. I can see why.

Is a relational shard even the right shape?

Sometimes not. Four cases come up constantly, and each has an engine built for it.

Reads and writes can also split apart entirely, into two different data models. That's CQRS, Command Query Responsibility Segregation. The idea: stop forcing one schema to be good at both jobs. Keep a write model shaped for correctness and fast writes (normalized tables, one source of truth), and maintain one or more separate read models shaped for the exact queries your screens make (denormalized, pre-joined, even living in a different engine like a search index). A change flows one way, a write updates the write model and emits an event; a subscriber applies that event to each read model. It's distinct from a plain read replica, which is the same schema copied; here the read side is a deliberately different representation.

Why it helps: each side scales and is optimized independently, your query screens never contort the write schema, and your writes never carry read-shaped baggage. Why it's dangerous: you've doubled your models and taken on the sync machinery between them. Martin Fowler, who did much to popularize it, warns against reaching for it by default; in his experience most broad applications of it added more complexity than they removed. Microsoft's architecture guidance names the core trade-off: the read side can lag the write side (it's eventually consistent by construction), and keeping them in sync is an ongoing engineering job, not a one-time setup. Reach for it on the one subsystem whose read and write shapes genuinely diverge, not across the whole app.

20 million users: what does the user actually need before you respond?

Axis: async work. Writes are partitioned now. So what is a user still waiting 45 seconds for? Ask the question honestly. For a clip upload, the user needs exactly one thing before you respond: the file is durably stored. Everything else (the transcode, thumbnails, moderation, notifications, the feed fan-out, search indexing) can happen after the response is already in their hands.

One line or many: queues and topics

The work is off the request path. Where does it go? It goes into a message broker, a durable middleman (SQS, RabbitMQ, Kafka) that holds units of work called messages until a worker (a separate process, scaled independently of your web servers) picks one up and does it. The web server's only job becomes: store the file, drop a message, respond. Everything slow now happens on the other side of that broker, decoupled in time from the user's request. There are two shapes of broker, and the difference is how many workers each message reaches.

Scroll sideways to see the whole diagram.

• A queue delivers each message to exactly one worker out of a pool (the "competing consumers" pattern). Add workers and they split the backlog between them, so the work goes faster but still happens once. That fits a transcode: the clip needs to be encoded a single time, and it doesn't matter which worker does it.
• A topic delivers a copy of each message to every subscriber (publish/subscribe). The upload service publishes one "clip uploaded" event; notifications, feed fan-out, and search indexing are three independent subscribers, each getting its own copy to process on its own schedule, at its own speed, failing and retrying without touching the others.

The rule of thumb: a queue splits work that must happen once; a topic broadcasts an event that several unrelated things must each react to. Many systems use both, a topic for the event, feeding a queue per consumer.

And here's the payoff for autoscaling: scale these workers on queue depth (messages waiting per worker), not CPU. The backlog is the amount of work waiting, so it's the most honest possible signal, if depth is climbing you're falling behind and need more workers; if it's near zero you can scale down. Compare that to CPU, which (as the earlier auto-scaler trap showed) can sit low while workers wait on I/O. It's the cleanest autoscaling signal in this whole post.

Duplicates are the normal case

What happens when a worker sees the same message twice? Plan for it, because it will. The property you need is idempotence: an operation is idempotent if doing it twice leaves the system in the same state as doing it once. "Set the thumbnail to X" is idempotent; "add $5 to the balance" is not. Amazon SQS standard queues say outright that a message may be delivered more than once and your application must be idempotent.

Can't the broker just promise to deliver each message once? Vendors advertise it, SQS FIFO offers exactly-once processing, Kafka has shipped exactly-once semantics since version 0.11, but read the scope carefully. Those guarantees cover duplicates introduced inside the messaging system's own boundary. They can't cover the dangerous case: a worker that performs a real side effect out in the world (charges a card, sends an email, writes to another database) and then crashes before it acknowledges the message. The broker never heard "done," so it correctly redelivers, and a second worker repeats the side effect. No broker can prevent that, because the side effect happened somewhere the broker can't see or undo.

So the honest, practical recipe is at-least-once delivery plus idempotent processing. Let the broker redeliver freely (at-least-once is easy and reliable), and make the handler recognize a message it has already processed and skip the duplicate. The two together add up to effectively-once, the real-world version of the guarantee marketing promises. The standard implementation is a dedupe table keyed by message id, written in the same transaction as the work:

def handle(msg):
    with db.transaction():
        first_time = db.insert_if_absent("handled_events", id=msg.id)  # unique key on id
        if not first_time:
            return                                # duplicate delivery: already done
        db.insert("thumbnails", clip_id=msg.clip_id, ...)
    queue.ack(msg)                                # ack only after the work is committed

Exactly-once is a property of a pipeline, not a promise about your side effects. Make the handler idempotent.

Order costs throughput

The very thing that made async work fast, spreading messages across many workers running in parallel, is what destroys their order. If two workers pick up "edit clip" then "delete clip" at the same instant, whichever finishes first wins, and you can't predict which. Order and parallelism pull in opposite directions: to guarantee A-before-B you have to process A and B on the same lane, one after another, which means no parallelism on that lane. That's why a partitioned log like Kafka guarantees order only within a single partition (one consumer reads it strictly in sequence), never across the whole topic.

So ordering has a throughput price, and SQS FIFO documents it: roughly 300 operations per second per API action, or 3,000 messages per second with batching, far below what an unordered queue sustains. Paying that across your whole stream would be ruinous. The way out is to make the ordered lanes narrow: guarantee order only per message group, a key like user-id or clip-id, so all messages for one clip are serialized, while different clips run fully in parallel. You get correctness where it matters (events for the same clip stay in order) and keep throughput everywhere else (a million clips, a million independent lanes).

The failure that never errors

Some messages never get processed even once, and nothing logs an error. To see how, you need one more concept: the visibility timeout. When a worker picks up a message, the broker doesn't delete it, it hides it from other workers for a set window and waits for the worker to send an acknowledgement ("done, delete it") when the work completes. If that ack never arrives within the window, the broker assumes the worker died and makes the message visible again for someone else. This is how at-least-once delivery survives crashes.

Which way you get the window wrong decides how bad the bug is:

• Timeout too short (shorter than your processing time): a nuisance, but survivable. The still-working message becomes visible again, a second worker starts it, and you get a duplicate, which your idempotent handler already absorbs.
• Ack before finishing (the genuinely dangerous one): a worker acknowledges the message first, then does the work, then crashes mid-way. The broker already deleted the message, so there's nothing left to redeliver. The work silently never happens, and no error is ever logged. The rule: ack only after the work is durably committed, exactly where queue.ack(msg) sits on the last line of the handler above.

For messages that fail repeatedly (a corrupt file that crashes every worker that touches it), the broker can "poison-pill" them into a dead-letter queue (DLQ) after a configured number of attempts, so one bad message doesn't block the line forever. But a DLQ only isolates failures; it doesn't tell anyone. In AWS's own documentation, alarming on it is a separate, optional step, so an unmonitored DLQ can quietly collect broken work for months. Put an alarm on its depth the day you create it.

When the database and the event disagree

This is the dual-write problem, and it's subtle because the happy path always works. Your upload service has to do two things: commit the clip row to its database and publish a "clip uploaded" event to the broker. Those are two separate systems with no shared transaction, so there's no way to make both happen atomically. Commit the row, then crash before publishing, and you have a clip that exists but that notifications, feed, and search never heard about. Publish first, then have the transaction roll back, and you've announced a clip that doesn't exist. Either way the database and the event stream now disagree, permanently, and nothing errored.

Scroll sideways to see the whole diagram.

The outbox pattern sidesteps it by refusing to do two writes to two systems at all. Instead of publishing to the broker directly, the service writes the event as a row into an outbox table in its own database, inside the same transaction as the business write. Now atomicity is free, it's one transaction to one database, so the clip row and its outbox row commit together or roll back together; they can't disagree. A separate relay process then reads unpublished rows from the outbox table and forwards them to the broker, marking each as sent. If the relay crashes, it just resumes, the events are safe in the table. Debezium implements this efficiently by tailing the database's change log (the same WAL/binlog that feeds replicas) rather than polling the table, so new outbox rows are picked up almost instantly with no query load.

Commit and publish together, or not at all. The trick is realizing you only ever have to commit, to one database, and let a relay turn durable rows into events.

When producers outrun consumers

When producers put work in faster than consumers take it out, the queue grows, and an unbounded queue feels like a safety valve but is really a trap. It doesn't prevent overload; it hides it, then delays it. Every message added behind a growing backlog waits longer before it's served, so latency climbs without limit even though nothing has technically failed. (Little's Law makes this precise: average wait time = queue length ÷ processing rate, so a queue that only grows produces a wait that only grows.) By the time the backlog is visible, you're serving work whose requester gave up long ago.

The discipline is backpressure: when a component can't keep up, it should push back, refuse new work early and explicitly, rather than accept it and quietly fall behind. Concretely, bound the queue. Google's SRE book suggests keeping it short relative to the thread pool, around half its size or less for steady traffic, so once it's full the server rejects new work fast instead of accepting what it can't finish. The Reactive Manifesto states the principle cleanly: an overwhelmed component shouldn't fail badly or drop work silently; it should signal upstream to slow down. And that signal isn't abstract, backlog per instance is a metric AWS builds directly into its autoscaling guidance, so "I'm full" and "add more workers" are the same number read by two different systems.

Signal upstream. Don't drop work silently. A fast, honest rejection beats an accepted request that dies waiting.

100 million users: why it still falls over on launch day

Axis: resilience. Every axis scales now. And yet, past a certain size, the machinery of scaling starts making outages worse. The failures in this chapter don't come from too little capacity. They come from systems reacting to trouble in ways that multiply it.

Retries are selfish

A service slows down. Its clients time out and retry, and those retries multiply the load on the one thing that's already struggling. AWS's Builders' Library is blunt about it: a timeout set too low can turn a slowdown into a complete outage, because every request starts retrying, and retries can deepen an overload and delay recovery.

This is not hypothetical. In AWS's own account of the October 2025 us-east-1 outage, once the triggering DNS problem was fixed, the EC2 subsystem that manages the physical hosts tried to re-establish leases on a huge backlog of hosts at once. The work timed out and was retried faster than it could complete, and the system fell into what AWS called congestive collapse. Engineers had to throttle incoming work and restart hosts to get it moving again, which stretched the incident across many services for hours.

The baseline everyone starts with is exponential backoff: after a failure, wait a bit before retrying, and double the wait each time (1s, 2s, 4s, 8s…) so a struggling service gets geometrically more breathing room. It's necessary but not sufficient, because it fixes how often one client retries without fixing the fact that all clients retry in sync. They all failed at the same instant, so they all back off on the same schedule and come back in the same synchronized waves, each wave still a spike that re-overloads the service. Two more fixes close the gap:

• Jitter. Add randomness to every backoff interval (wait a random time up to the backoff, not exactly it) so clients scatter instead of returning in lockstep. This is the single highest-leverage line in the whole retry story: Marc Brooker's simulations at AWS found full jitter cut the total work clients sent by more than half compared with plain exponential backoff, by dissolving the waves into a smear (exactly what the figure above shows).
• A retry budget. Backoff and jitter space retries out, but during a real outage even well-spaced retries are wasted load piled on something that's down. A budget caps the total: Google's SRE book is concrete, cap each request at three attempts, and let a client retry only while retries stay under about 10% of its outgoing traffic. When a large fraction of requests are failing, the budget is exhausted and retries simply stop, so a failing dependency doesn't get a self-inflicted traffic multiplier on top of its trouble.

And never blindly retry a non-idempotent write. A timeout doesn't mean the side effect didn't happen, it means you don't know whether it did; retrying a "charge the card" that actually succeeded charges it twice. Retry only operations you've made idempotent (see the async chapter), or carry an idempotency key the server can dedupe on.

The outage wasn't caused by the failure. It was caused by the retries.

One slow dependency, a whole chain down

This is cascading failure, and the mechanism is thread exhaustion. A server handles requests with a finite pool of threads (or connections). A thread that's waiting on a downstream call is not free, it's held, doing nothing, until that call returns. So when service C slows down and B calls C with no timeout, B's threads pile up waiting on C and are never returned to the pool. Once every thread in B is stuck waiting, B can't serve anything, not even requests that have nothing to do with C. Then A, calling B, exhausts its own pool the same way. One slow leaf takes down the entire tree above it, even though nothing actually crashed.

Scroll sideways to see the whole diagram.

The fixes go back to Michael Nygard's Release It!:

• Timeouts everywhere. Cap how long any call may wait, so a slow call fails fast and returns its thread instead of holding it indefinitely. A request with no timeout is a resource leak.
• Bulkheads. Named after a ship's watertight compartments, give each dependency its own separate thread or connection pool. Then when the pool for dependency C is swamped, it's only C's compartment that floods; calls to A's other dependencies draw from their own pools and keep flowing. One slow neighbor can no longer sink the whole ship.

Netflix's Hystrix library was built around exactly these ideas; it's in maintenance mode now, and Resilience4j is the common successor.

Timeouts don't add up. Budgets do.

"Timeouts everywhere" is the right instinct, and on its own it doesn't add up. Give each hop its own generous timeout, go three hops deep, and the worst case is the sum of all of them. The user gave up long before that. Every hop stayed inside its limit, and the request still took far too long.

So stop giving each hop a timeout and give the request a single deadline that travels with it, a technique called deadline propagation. The entry point stamps the request with "you have 1,000 ms total." Each service subtracts the time it has already used and passes the remainder down (gRPC propagates deadlines natively this way). By the last hop there's almost nothing left, so it fails immediately rather than starting work it can't possibly finish in time. The worst case is now the budget, not the sum of the hops.

Circuit breakers: knowing when to stop trying

A deadline fixes one request. It does nothing about the next thousand, all trying the same dead dependency and each dutifully waiting out its timeout before failing. That's thousands of threads and seconds wasted on something you already know is down. A circuit breaker is a small piece of state that sits in front of a dependency and remembers how it's been behaving, so the caller can stop trying instead of rediscovering the failure one request at a time. Picture a physical breaker switch. Closed: current flows, calls go through, failures are counted. Once failures cross a threshold, the lever flips open: every call fails instantly without even touching the dependency, which both protects the caller's threads and takes load off the struggling dependency so it has room to recover. After a cooldown the breaker goes half-open and lets a single trial request through to test the water: success closes it (back to normal), failure reopens it and restarts the cooldown. It's the difference between a crowd rattling a locked door for an hour and one person checking it once a minute.

Shed load to protect goodput

One failing dependency is contained. What happens when the whole system is simply over capacity? The instinct is to be generous, accept every request and try to serve them all. That's exactly wrong, because past capacity, accepting more work doesn't serve more users, it just makes every request slower until none finishes in time. Load shedding is the discipline of deliberately rejecting excess requests (cheaply, with a fast 503) so the ones you keep actually complete.

The reason this is counterintuitive is that the usual scoreboard lies. AWS's Builders' Library gives a sharp example: shed a big slice of your traffic on purpose and your median latency suddenly looks fantastic, because the requests you rejected failed so fast they dragged the number down. Latency is measuring the wrong thing. The number that matters is goodput: not how many requests you handled (that's throughput, and it counts the useless failures), but how many you handled well, completed correctly within their deadline. A request served so slowly the user already left is throughput without goodput: work done, value zero.

Google tags every request with one of four criticality levels and sheds the least important first, automatically. Drop ten percent and the other ninety stay healthy. Accept everything and you keep none of it.

Rate limiting: who gets shed first

Rate limiting decides which requests get shed: it caps how many each client (per API key, per user, per IP) may make in a window, so one heavy caller can't consume the capacity meant for everyone. The elegant part is encoding two different limits, a sustained rate and a burst allowance, in one simple mechanism: the token bucket. Picture a bucket that fills with tokens at a fixed rate and has a maximum size. Each request must take one token to proceed; no token, no service (HTTP 429 Too Many Requests). The refill rate is your sustained limit (say 2/second forever); the bucket size is your burst allowance (say 8 saved up). Stripe's production limiter deliberately lets tokens accumulate, so a genuine burst, a spike during a live moment, can spend the saved tokens and get through, while a client that hammers forever is still held to the refill rate.

The common alternative is the sliding window. A fixed window (e.g. "100 requests per minute, reset on the minute") is easy but lets a client send 100 at 11:00:59 and 100 more at 11:01:00, 200 in two seconds, right at the boundary. The sliding window fixes that by keeping two counters, the current window and the previous one, and blending them by how far into the current window you are, producing a smooth estimate of the recent rate with no boundary spike. Cloudflare measured that approximation against real traffic and found it wrong on only about 0.003% of 400 million requests. The trade-off is clean: the bucket is simpler and more tolerant of deliberate bursts; the window is smoother and harder to game, at the cost of a little more state. Either way, per-tenant limits turn a noisy neighbor (one tenant's runaway traffic) from a shared outage into a 429 for that one customer.

Decide what's allowed to fail

Not every feature deserves the same protection. When the system is under real pressure, some things matter more than others, and the time to decide which is now, not during the incident. For Reelo:

Decide what is allowed to fail before load makes that decision for you.

Global: surviving the loss of a whole region

Axis: resilience, across geography. Your system survives being at capacity in one region. Whether it survives the region itself going away is a different question.

You're probably already spread across availability zones: separate buildings with their own power and network. Lose one and the others keep serving. But every one of those zones sits inside one region, and a region-wide event (a control-plane bug, a bad config pushed everywhere at once) takes all of them at the same moment. Multi-AZ was never designed to cover that.

Scroll sideways to see the whole diagram.

So the second region has to be genuinely far away, which brings back the number from the CDN section: a couple of hundred milliseconds to the other side of the world, and it doesn't move. That makes a second region a survival decision, not a performance one.

What a second region actually costs

Compute replicates easily: deploy the same service in a second region and it just works. AWS's Well-Architected disaster recovery guidance lays out four strategies in rising order of cost and complexity.

The reason active-active is hardest: two regions writing to the same record will, eventually, conflict.

Replicating compute is trivial. Replicating state is the entire problem.

When the regions can't talk

Both regions serve traffic now. What happens the instant they can't reach each other? The answer is the CAP theorem, and it's worth stating precisely because I'd seen it mangled everywhere, and had it wrong myself before I slowed down and read the actual statement. A distributed system would like three properties: Consistency (every read sees the latest write), Availability (every request gets a non-error response), and Partition tolerance (the system keeps working when the network between nodes drops messages). Eric Brewer proposed in 2000, and Seth Gilbert and Nancy Lynch proved in 2002, that you cannot have all three at once.

The key insight is that a network partition is not optional, between two regions 200 ms apart, the link will sometimes break, so P is a given, not a choice. The real choice only appears during a partition, and it's a forced choice between C and A. Think it through: the two regions can't talk, and a write lands on region A. Either A accepts it (now region B, still serving reads, hands out stale data, you kept Availability and gave up Consistency), or A refuses it until it can confirm with B (now that user gets an error, you kept Consistency and gave up Availability). There's no third door while the link is down.

Pick "keep availability" and you've accepted that two regions can write the same record during a partition, so you need a rule to reconcile them afterward. The simplest, and DynamoDB global tables' default, is last-write-wins (LWW): when two conflicting writes to the same item meet, compare their timestamps and keep the more recent one. It's appealingly simple, but notice what it does, the older write is silently discarded, with no error and no merge. That's perfectly fine for "where in the video this viewer last paused" (you only ever wanted the latest value anyway) and catastrophic for a creator's payout balance or the remaining stock of a limited merch drop, where the discarded write was real money. (AWS has since added a strongly consistent multi-region option, which makes the opposite CAP trade, choosing C over A.) The lesson isn't that one choice is right; it's that each piece of data needs its own choice, and LWW decides for you unless you decide first.

Cells: bounding the blast radius

There's a different reason large companies split into cells. Picture a grid of identical, self-contained copies of the whole stack, each serving its own slice of customers. When one cell gets a bad deploy and fails, every other cell, running its own independent copy, is unaffected.

The bill: what all of it costs

Axis: dollars, at every rung. Cells, regions, replicas, queues: you've built a lot of machinery, and none of it is free. Cost per thousand requests, the number you started tracking at 100 users, is what keeps this visible instead of hidden inside one big monthly bill.

The industry trend is going the wrong way. Flexera's 2026 State of the Cloud report put wasted cloud spend, as estimated by the respondents themselves, at 29 percent, up from 27 the year before and the first increase in five years. And studies of enterprise fleets have long found typical servers doing only a small fraction, often in the 5 to 15 percent range, of the work they're capable of. That's capacity paid for and never used.

Scaling that makes each user five times more expensive isn't scaling. It's spending.

Match the pricing to the shape of the load

Most of the fleet is paid for and idle. The lever is to buy each slice of your load the way it actually behaves.

Managed services are the same trade in a different currency: you pay more per unit to spend less engineering time running the thing yourself. Whether that's a good deal depends on what your engineers would otherwise be doing.

Storage has a shape too

Look at when your data is actually read. A clip is watched hard in its first days and then almost never again, yet by default you pay to keep every one equally fast. Most of what you store is cold, sitting on your most expensive tier.

A lifecycle policy is a guess about access patterns, so check it against how your data is really read before a retrieval bill tells you the guess was wrong.

The line people find last: data transfer

Data coming in is generally free. Data going out is not. Every arrow in your architecture that crosses a boundary has a price on it, and none of them look any different on the diagram.

Scroll sideways to see the whole diagram.

Within a zone is generally free, crossing zones costs you, and out to the internet costs the most. There's no single published rate, because it depends on the service and the region. The useful move is to look at your own diagram and ask which arrows cross a boundary. The chatty pair of services that ended up in two different zones is the one that would have surprised me.

The part the bill doesn't cover

One thing is left: a region going away and everything having to move. DNS TTLs bound how fast traffic can actually be redirected, and your health checks decide when a region is unhealthy enough to fail away from. Be honest about the risk here. A failover runbook that has never been executed is, by definition, untested. Rehearse it on a calm day, the same way you found your ceiling with a load test at 100 users.

2 a.m.: the playbook

Pull all the way back. This is the same architecture from the top of the page.

Scroll sideways to see the whole diagram.

At the start it overwhelmed me. Now I can look at it and name every box, say what problem it solves, and say what it costs, which still feels a little unreasonable for something I opened by trying to close the tab. So how would you actually use any of this at two in the morning, when something is slow and you don't know where to look? Find out which of the four axes is actually saturated, then work its list from the top. Each list starts with the cheapest, most reversible fix.

Compute

1. Scale vertically
2. Make the app stateless
3. Scale horizontally
4. Load-balance and autoscale on the real signal

Reads

1. Fix the query: indexes, N+1s, keyset pagination
2. Cache it
3. Add read replicas
4. Push it to a CDN

Writes

1. Pool connections and batch writes
2. Split by function
3. Shard by key

Async

1. Move it off the request path
2. Make workers idempotent
3. Scale on queue depth

The whole ladder, one more time

Every rung was a different bottleneck and a different fix, and none of them were needed a rung early. That's the thing I'll take away from the diagram I started out wanting to close: it wasn't designed, it was earned, one red bar at a time.

Sources and further reading

1. Google SRE book: Monitoring Distributed Systems (the four golden signals)
2. Dean and Barroso, The Tail at Scale (CACM, 2013)
3. Brendan Gregg, The USE Method
4. Datadog: 100x faster Postgres performance by changing 1 line
5. Sentry: Eliminating N+1 queries
6. The Twelve-Factor App: Processes (on sticky sessions)
7. Google Cloud: Bringing Pokémon GO to life
8. GitHub: Mitigating replication lag with freno
9. Werner Vogels, Eventually Consistent (2008)
10. Nishtala et al., Scaling Memcache at Facebook (NSDI 2013)
11. Dan Luu and Yao Yue, A decade of major cache incidents at Twitter
12. Markus Winand, Use The Index, Luke: Insert
13. DeCandia et al., Dynamo: Amazon's Highly Available Key-value Store (2007)
14. Martin Fowler, CQRS
15. AWS: Summary of the October 2025 us-east-1 service disruption
16. AWS Builders' Library: Timeouts, retries and backoff with jitter
17. Marc Brooker, Exponential Backoff and Jitter
18. Google SRE book: Addressing Cascading Failures
19. Google SRE book: Handling Overload
20. AWS Builders' Library: Using load shedding to avoid overload
21. Stripe: Scaling your API with rate limiters
22. Cloudflare: How we built rate limiting capable of scaling to millions of domains
23. The Reactive Manifesto
24. Flexera 2026 State of the Cloud Report