The Three Pillars Are Not the Point. Cardinality Is, and It Is What You Get Billed For.

Logs, metrics and traces is a taxonomy of data formats, not of capability. Having all three does not mean you can answer a question you did not anticipate. What lets you do that is high-cardinality context on every request, which is also precisely what your observability vendor prices on.

Tech Talk News Editorial9 min read
ShareXLinkedInRedditEmail
The Three Pillars Are Not the Point. Cardinality Is, and It Is What You Get Billed For.

Key takeaways

  • Monitoring answers known unknowns, meaning "is the thing I predicted happening", while observability answers unknown unknowns, meaning "why is this specific request slow when nobody predicted this failure".
  • The three pillars framing describes data formats rather than capability, so having logs, metrics and traces does not establish that you can investigate a novel problem without shipping new code.
  • The three-pillars vocabulary spread around 2020 as vendors entered the space, and the definition broadened to the point where it could describe almost any tooling.
  • The functional unit of observability is the arbitrarily wide structured event: one record per request carrying every relevant attribute, including user ID, build ID, request ID, tenant, region and feature flag state.
  • Cardinality is the number of distinct values a field can take, and high-cardinality fields such as user ID are exactly what make a novel question answerable.
  • The main cost drivers are the number of tools, the cardinality of the data and its dimensionality, so the standard mitigations of sampling and cardinality limits reduce the fidelity that made the data useful.

Ask a team whether they have observability and you will usually be told about the three pillars: logs, metrics, traces. All three present, therefore observable.

That reasoning does not hold, because the three pillars are a taxonomy of data formats and observability is a capability. Owning three shapes of data tells you nothing about whether you can answer a question nobody thought to ask in advance, which is the only test that matters at three in the morning.

Charity Majors, who has argued this longer and more bluntly than anyone, puts it as: there are no three pillars, the pillars are nonsense. The framing spread around 2020 as vendors moved into the space, and the definition broadened until it could describe almost any tooling.[1]

Known
Unknowns monitoring covers
Unknown
Unknowns observability covers
Wide event
The functional unit, one per request
Cardinality
What makes it work, and what you are billed on

The Distinction That Actually Separates Them

The cleanest formulation, and it is worth memorising because it settles most arguments: monitoring handles known unknowns; observability handles unknown unknowns.[2]

Monitoring answers “is the thing I predicted happening?” You decided in advance that CPU above 80% matters, so you built a dashboard and an alert. That is genuinely valuable and it is not observability, because it can only ever tell you about failures you already imagined.

Observability answers “why is this specific requestslow, when nobody predicted this?”[2] Nobody built a dashboard for it. There is no alert. It is happening anyway, and the question is whether you can investigate it right now or whether you have to ship instrumentation first and wait for it to recur.

That last clause is the practical test. If diagnosing a novel problem requires a deploy, you have monitoring.

If investigating an unexpected failure requires shipping new instrumentation and waiting for it to happen again, what you have is monitoring.

The Unit That Makes It Work

If not pillars, then what? The functional unit is the arbitrarily wide structured event: one record per request, carrying every attribute that was true at the moment it happened.[3]

User ID. Build ID. Request ID. Tenant. Region. Feature flag state. Whatever else was in scope.[3] Not three separate systems you correlate afterwards by squinting at timestamps, but one wide row you can slice by any of those fields after the fact.

Why that shape rather than metrics: because metrics are pre-aggregated, and aggregation is lossy in exactly the direction that hurts. A p99 latency number tells you something is slow for someone. It cannot tell you that the someone is customers on build 4823, in eu-west, with one flag enabled, because that information was averaged away at write time. You cannot recover a dimension you did not keep. The distinction between throughput and responsiveness in latency versus bandwidth is the same lesson at a different layer: the aggregate hides the thing you are hunting.

Cardinality, and the Bill

Cardinality is the number of distinct values a field can take.[3]Region is low cardinality, maybe a dozen values. User ID is high cardinality, one per user.

High-cardinality fields are the ones that make novel questions answerable, because narrowing to which requests are failing requires a field with enough distinct values to isolate them. Low-cardinality data supports low-resolution answers.

Now the part that deserves to be said plainly, because vendor documentation puts it delicately. The main cost drivers in observability tooling are the number of tools, the cardinality of the data, and its dimensionality, meaning how much context you keep per record.[4]

Those are the same three properties that make the data useful. You are billed in direct proportion to the capability you are buying, which means the pricing model runs against the reason you bought the tool.

Why this matters

The standard mitigations make this worse rather than better. At scale teams reach for sampling and cardinality limits, and both reduce the fidelity and utility of the data itself.[4] Note which requests sampling discards: the rare ones. The rare request is also the one you most need during an incident, so aggressive sampling optimises your bill by removing exactly the evidence you will go looking for.

None of which means do not sample. It means sample deliberately, keeping errors and slow outliers at a much higher rate than successful fast requests, rather than uniformly at 1% because that was the default. The decision is which questions you are willing to become unable to answer, and it should be made explicitly rather than by a config value nobody revisited.

What To Do With This

Test the capability, not the inventory. Pick a real question you could not have predicted, such as which customers saw elevated errors after the last deploy, and try to answer it with what you have. If you cannot, adding a fourth log aggregator will not help.

Instrument wide, not deep. One event per unit of work with many attributes beats many narrow log lines you later try to reassemble. Put the identifiers on it at the time, because you cannot add a field retroactively to an event that has already been written.

Keep build ID and feature flag state.These are the two highest-value fields most teams omit, and they are the ones that turn “something changed” into “this change did it”. The second matters more than it looks, because a flag flip changes production without a deploy, so an incident where nothing was deployed can still have a cause.

Sample by value, not uniformly. Keep every error and every slow request. Sample the boring successes hard. Your bill barely notices and your incidents get much shorter.

Remember what this is for. Observability is how you find out that a dependency has become intermittently slow in a way that never trips a threshold, which is exactly the failure a circuit breaker cannot detect and a bulkhead only contains. The tooling and the resilience patterns are answers to the same problem from opposite ends.

Takeaway

Logs, metrics and traces are data formats, not a capability, and holding all three proves nothing. The test is whether you can investigate a failure nobody predicted without shipping code first. That requires wide structured events carrying high-cardinality context, which is also precisely what your vendor bills on, so the economics push you toward sampling away the rare requests you will most need. Sample by value instead, and keep build ID and flag state on everything.

Sources and further reading

  1. 1.PrimaryCharity Majors on Screaming in the Cloud, "Shifting from Observability 1.0 to 2.0". Source for the argument that the three pillars are not a definition of observability, and that the vocabulary broadened as vendors entered the space around 2020. Her own writing is at charity.wtf.
  2. 2.Reporting"Monitoring vs observability: which catches unknown failures first". Source for the known-unknowns versus unknown-unknowns distinction and the framing of the questions each answers.
  3. 3.PrimaryHoneycomb, "So you want to build an observability tool...". Source for the arbitrarily wide structured event as the unit, the attributes it should carry, and the definition of cardinality.
  4. 4.ReportingThe New Stack, "Beyond the 3 pillars of observability". Source for the cost drivers being tool count, cardinality and dimensionality, and for sampling and cardinality limits reducing the fidelity of the data.

Frequently asked questions

What is the difference between monitoring and observability?
Monitoring covers known unknowns: you decided in advance what to watch, and it tells you whether that thing is happening. Observability covers unknown unknowns: it lets you ask a question nobody anticipated, such as why one particular customer's requests are slow, without deploying new instrumentation first.
Are the three pillars of observability real?
They are a real description of data formats and a poor description of capability. Logs, metrics and traces are three shapes data can take, but possessing all three says nothing about whether you can investigate a failure mode you did not predict, which is the actual test. The framing became popular around 2020 as vendors entered the market and the definition broadened accordingly.
What is cardinality in observability?
Cardinality is the number of distinct values a field can take. Region is low cardinality with perhaps a dozen values, while user ID is high cardinality with as many values as you have users. High-cardinality fields are what let you narrow an investigation to the specific requests that are failing rather than an aggregate that hides them.
Why does high cardinality matter so much?
Because aggregates hide the failures you are hunting. A p99 latency figure tells you something is slow for someone but not who, and if the answer is "customers on one build, in one region, with one feature flag enabled", you can only reach it by filtering on fields with many distinct values. Without that, every investigation ends at a graph that looks fine.
Why do observability tools get so expensive?
Because the main cost drivers are the number of tools, the cardinality of the data and its dimensionality, which are the same properties that make the data useful. You are billed in proportion to the context you keep, so the pricing model runs directly against the reason you bought the tool.
Does sampling solve the observability cost problem?
It controls the bill while degrading the capability, which is a trade rather than a solution. Sampling and cardinality limits reduce the fidelity and utility of the data itself, and the rare request you most need to investigate is statistically the one most likely to have been discarded.

Written by

Tech Talk News Editorial

Computer engineering background. Writes about software, AI, markets, and real estate, and the places where the three meet.

More about the author
ShareXLinkedInRedditEmail