Building systems that scale

Scale Is Not Traffic

Most teams hear "scalable" and picture load. But systems rarely fall over because too many people arrived at once. They fall over because a decision made in week two turned out to be expensive to reverse in month twenty.

Scaling is really about which decisions you can still change cheaply. A system scales well when adding the tenth feature costs roughly what the third one cost, and when the second customer does not require a rewrite.

The Decisions That Are Expensive To Reverse

A small number of choices are structural, and they are worth slowing down for:

• The data model. Renaming a field is easy. Discovering that an entity you modelled as belonging to a user actually belongs to an organisation is not.
• Tenancy. Whether rows belong to an account, and whether that is enforced at the query layer or hoped for in the controller.
• Identity. Who a user is, what they can act on, and whether permissions are a property of the row or a property of the session.
• Time. Storing local times without zones is a decision you will pay for the first time someone travels.

Almost everything else can be changed later without drama. Spend your early caution here and be relaxed about the rest.

Async All The Way Down, Or Not At All

A single blocking call inside an otherwise asynchronous request path will quietly serialise everything behind it. If the framework is async, the database driver has to be async too, and so does the HTTP client used to call that third-party API.

Size the connection pool deliberately rather than accepting the default. Workers multiplied by pool size has to stay under whatever the database will actually accept, and the moment it does not you get failures that look like application bugs and are not.

Every Cache Is A Promise

Caching is usually introduced as a performance fix and then quietly becomes a correctness problem. The question is never "is this faster" but "how wrong is this allowed to be, and for how long". Answer it for each cached thing.

Some data can be a minute stale and nobody notices. Some data being one second stale means a customer sees another customer's total. Those are not the same decision and should not share a default.

Background Work Will Run Twice

Any job that can be retried will eventually be retried, and any job that can run twice eventually will. Design for it. An operation that is safe to repeat can be retried aggressively, so you can be relaxed about transient failures instead of building elaborate machinery to prevent them.

Give jobs an identity, record that the work happened, and check before repeating it. This is far less effort than reconciling duplicate charges later.

Observability Before You Need It

The cheapest time to add structured logging, request identifiers and basic metrics is before the incident, because during the incident you cannot deploy the instrumentation that would have told you what is wrong.

Log to standard output as structured records and let the platform handle shipping. Carry a request identifier through every layer so one user's bad afternoon can be reconstructed from a single search. Alert on latency, error rate and pool saturation rather than on uptime, because a system that is up and taking thirty seconds per request is down as far as anyone using it is concerned.

Deploys Are Part Of The Design

A system that can only be released at midnight is not finished. Migrations should run as a separate step before the new code goes out, and should stay compatible with the version still serving traffic for the length of the rollout. Add a column, deploy the code that writes it, and only then remove the old one, in three releases rather than one brave afternoon.

What To Skip Early

Sharding, service extraction, custom queues and elaborate caching layers all solve real problems, but usually not the problem you have. A single well-indexed database will carry a product much further than most architecture diagrams suggest.

Build the boundaries in code so they can be pulled apart later. Do not pull them apart until something is actually hurting. The system that scales is usually the boring one that was easy to change.