Describing Load Without Lying

Before you can say whether a system scales, you have to say what is growing. That sounds obvious, and it is routinely skipped, which is why so many architecture discussions consist of two people confidently disagreeing about different systems.

Requests per second. Reads per write. Active users. Rows in the table everyone joins against. Cache hit rate. Which of these matters depends entirely on the system, and picking wrong sends you optimising something that was never the problem.

The example worth internalising

Consider a social feed. Two operations:

• Post a message. Perhaps 5,000 per second at peak.
• Read your home timeline — the merged, time-ordered posts of everyone you follow. Perhaps 300,000 per second.

There are two obvious designs.

Design A: do the work on read. Store posts in a table. When someone opens their timeline, look up who they follow, fetch those people’s recent posts, merge, sort, return.

SELECT p.* FROM posts p
  JOIN follows f ON f.followee_id = p.author_id
 WHERE f.follower_id = :me
 ORDER BY p.created_at DESC
 LIMIT 50

Writes are trivial — one insert. Reads do all the work, and there are sixty times more of them.

Design B: do the work on write. Keep a materialised timeline per user, like a mailbox. When someone posts, push that post into the timeline of every follower. Reading is then a single sequential read of an already-prepared list.

Reads become trivial. Writes become expensive — and here is the point:

An average user with 200 followers costs 200 writes per post. Fine. A celebrity with 30 million followers costs 30 million writes for one post, and they are posting during a live event, along with fifty other celebrities.

The average follower count tells you almost nothing. The tail of the distribution is the entire engineering problem.

Averages hide the number that hurts

The follower story generalises. When you describe load with a single average, you are asserting that the distribution is boring. It usually isn’t.

• “Average 500 requests per second.” If those arrive in a 5-second burst at the top of every minute, your real peak is 6,000/s and your capacity plan is wrong by an order of magnitude.
• “Average 40 rows per query.” If one customer has 400,000 rows and the query is unpaginated, you have a time bomb with a customer’s name on it.
• “Average cache hit rate 95%.” If misses cluster — a deploy, an eviction storm, a cold shard — then during that minute the hit rate is 0% and the database receives 20× its normal traffic.

Doing this for your own system

The practical exercise takes ten minutes and is worth doing honestly:

1. List the operations. What can the system be asked to do? Keep it to the handful that matter.
2. For each, ask what makes one call expensive. Not “how many calls” — what makes an individual one hard. Number of rows touched? Fan-out? Payload size?
3. Find the distribution, not the average. For each of those quantities, what does the p99 look like? Who is the biggest customer, and what happens when they do the normal thing?
4. Ask the doubling question. For each candidate parameter: if this doubled, what breaks first? If the answer is “nothing, we’d add a machine,” it is not your bottleneck.

The output is one or two numbers you can put in a document and argue about concretely — which is the whole point, because “it needs to scale” cannot be argued about at all.

What to take away

Load is not one number and it is rarely an average. Find the quantity whose doubling would force a redesign, look at its distribution rather than its mean, and write it down. The next lesson does the same for the other half of the sentence: what “performance” means once load is defined.