What Is Caching and Why Everything Uses It

There is a running joke among engineers that goes: there are only two hard things in computer science, cache invalidation and naming things. It is funny because it is true, and it is true because caching is everywhere. It is quietly holding up almost every fast thing you touch on a computer, and the moment it goes wrong it is one of the most maddening bugs to track down.

Here is the whole idea in one sentence. Caching means storing the result of some expensive work so you can reuse it instead of doing the work again. That is it. Everything else, the layers, the acronyms, the TTLs, is just detail on top of that one move: do the hard thing once, keep the answer, hand it back cheaply forever after.

The core move: do it once, reuse it forever

Imagine a function that takes two seconds to run. Maybe it queries a database, calls three other services, and does some math. The first time someone asks for the answer, you have no choice but to spend the two seconds. But if the answer will be the same for the next thousand people who ask, why pay two seconds a thousand times? Store it. Hand back the stored copy. Now the next thousand requests take a millisecond each.

That is the trade at the heart of every cache: you spend a little memory to save a lot of time. Memory is cheap and time is expensive, so it is usually a great trade. The way I think about it, caching is the closest thing computing has to a free lunch, right up until the moment the stored answer stops being correct. Then the lunch gets very expensive.

Hits, misses, and the hit rate that decides everything

Two words carry most of the vocabulary here. A cache hit is when the thing you asked for is already sitting in the cache, so it comes back instantly. A cache miss is when it is not there, so the system has to go do the full expensive work, and usually store the result on the way out so the next request hits instead.

The number that matters is the hit rate, the share of requests that are hits. A cache with a 95% hit rate means 19 out of 20 requests never touch the slow path. That is the entire game. A cache with a 10% hit rate is not helping you, it is just adding a layer of overhead and a new place for bugs to hide. Before you add a cache, the honest question is not “can I cache this,” it is “will the hit rate actually be high.”

Those rough numbers are why caching exists at all. Each layer down the stack is dramatically slower than the one above it, so keeping hot data as high up as possible is worth enormous effort.^[1] A cache is fundamentally a way to pull data up into a faster tier and keep it there while it is being used.

Caching happens at every layer, whether you know it or not

The reason caching feels abstract is that it is not one thing. It is a pattern repeated at every level of a computer, and most of the layers are invisible to you.

Start at the bottom. Your CPU has tiny caches called L1, L2, and L3, sitting physically closer to the cores than main memory does. Fetching from RAM is slow in CPU terms, so the processor keeps recently used data and instructions in these on-chip caches. This is happening billions of times a second while you read this sentence, and you will never see it.

Go up a level. Your browser caches images, scripts, and stylesheets on disk so the second visit to a site does not re-download everything. This is governed by HTTP caching headers like Cache-Control and ETag, which let a server tell the browser how long a file is good for and how to check whether it changed.^[2] It is the reason a site loads instantly the second time and crawls the first.

Up again. A CDN caches copies of a website's files in data centers around the world, so a visitor in Tokyo gets served from a nearby machine instead of waiting on a server in Virginia. A CDN is, at its core, a giant geographically distributed cache sitting in front of the origin.^[3] Then inside the application, a database query result or a rendered page gets stored in something like Redis, an in-memory data store fast enough to answer in well under a millisecond.^[4] And even DNS, the system that turns a domain name into an IP address, is cached at nearly every hop so your computer is not asking “where is example.com” from scratch every single time.

TTL: the timer that trades freshness for speed

If a cached copy could live forever, it would eventually be wrong, because the real data underneath it changes. So caches attach an expiry to entries, and the standard name for it is TTL, time to live. A TTL of 60 seconds says “this copy is good for a minute, after that go get a fresh one.”

TTL is the main dial you turn, and it is a genuine trade-off with no free setting. A short TTL keeps your data fresh but causes more misses, which means more trips to the slow path and less benefit from the cache. A long TTL is faster and cheaper but risks handing people outdated information. There is no universally correct value. The right TTL depends entirely on how fast the underlying data changes and how much a stale answer actually costs you.

A stock price might tolerate a one-second TTL at most. A user's profile photo could happily cache for a day. A blog post that never changes could cache for a year. Same mechanism, wildly different values, and picking them well is most of the craft in real-world caching.

The two hard problems: invalidation and staleness

Now the part the joke is about. Storing data in a cache is trivial. The hard problem is knowing when the stored copy has gone bad and getting rid of it at the right moment. This is called cache invalidation, and it is hard for a fundamental reason: a cache is a second copy of the truth, and the instant the real data changes, that copy is potentially lying.

There is no reliable universal signal that says “this specific cached entry is now outdated.” So you fall back on imperfect proxies. You set a TTL and accept that data can be stale for up to that long. Or you manually purge entries when you know something changed, which means you have to remember every place a piece of data got cached, and miss one and you serve stale data anyway. When you see a website where you updated something but the old version keeps showing up, you are looking at a cache invalidation bug in the wild.

Stalenessis the other side of the same coin: serving data that is technically expired or simply wrong because the cache has not caught up. A little staleness is fine for a blog. It is a disaster for an inventory count that lets you sell something you no longer have. The engineering question is never “can we avoid staleness entirely,” because you usually cannot. It is “how much staleness can this particular data tolerate.”

When caching adds more risk than value

Here is the opinion. Caching is treated as an automatic win, and it is not. A cache is a second source of truth you now have to keep in sync with the first, and that is real, permanent complexity. There are three cases where I think reaching for a cache is a mistake.

First, when the data changes constantly. If your hit rate is going to be low because entries expire almost as fast as you write them, the cache is pure overhead. You get the complexity and none of the speed. Second, when correctness matters more than speed. For a bank balance, an inventory count, or a permissions check, serving a stale value is worse than serving a slightly slower fresh one. A naive cache in front of that kind of data is a bug generator. Third, and most common, when the system is not actually slow. People add caches preemptively to systems that answer in five milliseconds, then spend weeks debugging staleness to shave off a millisecond nobody would have noticed. That is a bad trade.

This connects to how you think about database architecture in general. A cache is often a bandage on a slow query or a bad access pattern. Sometimes the right fix is not a cache in front of the database, it is a better index, a denormalized table, or a rethink of the query. Cache the thing that is genuinely expensive and genuinely reused. Do not cache to avoid understanding why something is slow.

What I'd do

My rule is boring on purpose. Do not add a cache until you have measured that something is actually slow and confirmed the data is read far more than it is written. Those two conditions, real slowness and a high read-to-write ratio, are what make caching pay off. If either one is missing, you are adding a liability, not a feature.

When you do cache, be deliberate about staleness up front. Decide how stale each piece of data is allowed to be, set the TTL to match, and write down how the cache gets invalidated when the underlying data changes. That last part is the one everyone skips and everyone regrets. The cache itself will work perfectly on day one. It is the invalidation you forgot to think about that pages you at 2am six months later. Caching is one of the highest-leverage tools in computing, but only when you respect the fact that you have created a copy of the truth, and copies of the truth have a way of drifting apart.