Somebody on your team is going to add a label this month. It will be a reasonable label. It will make a dashboard more useful, it will answer a question somebody asked in a retro, and it will pass review without comment.

Three weeks later your Prometheus will be getting killed by the kernel every few hours.

This is the most common way monitoring stacks fall over, and the thing that makes it interesting is not the technical mechanism. It is that the person who creates the cost is standing somewhere they cannot possibly see it.

Every combination is a series

Prometheus stores each unique combination of label values as its own time series. Not each metric. Each combination.

A metric with a status label taking five values and a method label taking four gives you twenty series. Add an endpoint label with fifty values and you have a thousand. The multiplication is the entire problem, and it stays polite right up until one of those labels is unbounded.

Unbounded means the set of possible values is not fixed by anything. User IDs. Email addresses. Session tokens. Raw URL paths with identifiers in them. Container IDs. Commit SHAs. Each new value creates a new series that lives forever in the index, and there is nothing in the system that pushes back.

Here is where the asymmetry comes in. In staging you have ten users and five endpoints, so a user_id label produces fifty series and looks completely harmless. The same label in production, against a million users over ninety days, produces something in the millions. The developer who added it saw the staging number. Nobody sees the production number until the memory graph does something alarming.

The arithmetic is unforgiving

Prometheus keeps its head block, meaning the most recent data, entirely in memory. That design is why it is fast. It is also why cardinality hurts in a way that disk-backed systems do not.

Each active series needs roughly three to four kilobytes of RAM for the series itself plus its index entries. As Alexandre Vazquez sets out in a detailed breakdown, an instance carrying a million active series will typically sit at four to six gigabytes just for the head block, before any query runs.

There is no low-memory mode to fall back on. Prometheus does not offer a setting that trades RAM for slower queries. On memory, it is all or nothing.

What it actually looks like when it happens

It rarely announces itself as a cardinality problem. It announces itself as three or four separate problems that look unrelated.

Memory on the Prometheus process climbs past what anyone expected. A box that ran comfortably at four gigabytes starts touching twelve, then twenty. Then the kernel intervenes and the process is OOM-killed.

The restart looks like a fix. Memory is low, everything responds, and somebody closes the ticket. Then Prometheus rebuilds the same head series from the write-ahead log and from incoming scrapes, and hits the same ceiling within hours. That cycle is the clearest signature of the problem, and it gets misread as an under-provisioned instance more often than anything else I have seen.

Alongside it, PromQL that was fine at a hundred thousand series becomes unusable at a million. Grafana panels start timing out. Alert evaluation falls behind its own schedule, which means Alertmanager begins sending alerts late, or twice.

There is a subtler failure worth knowing about. If you have set a sample_limit on a scrape target, tripping it does not drop the excess samples. The entire scrape fails, and the target reports up=0. Your service looks down when it is fine. Last9's write-up recommends alerting on prometheus_target_scrapes_exceeded_sample_limit_total rather than discovering this during an incident, which is good advice.

The timing is the cruel part

Cardinality problems surface under load, and load is when things break. So the sequence tends to be: an incident begins, traffic and pod churn push series count up, Prometheus becomes slow or dies, and you lose your monitoring at the exact moment you need to look at it.

Then the standard mitigations make it worse in a specific way. Under memory pressure teams drop metrics or shorten retention. Both work. Both also delete the data you were about to use to understand what happened. You buy stability by destroying the evidence.

This is why I have stopped filing cardinality under performance. It is an availability problem, and the thing whose availability is at stake is your ability to see anything at all.

Finding the offender

When series count is climbing, the TSDB status page in the Prometheus UI is the fastest first look. It lists the metrics with the highest series counts directly, which usually identifies the culprit in under a minute.

For queries, the one I reach for first counts series per metric name:

topk(10, count by (__name__)({__name__=~".+"}))

Then, once you have a suspect metric, find which label is doing the damage:

count(count by (label_name) (suspect_metric_name))

The metric to watch continuously is prometheus_tsdb_head_series. A healthy instance shows it plateauing. A cardinality problem shows it climbing steadily and never levelling off, which is a much better alarm than memory usage because it fires days earlier.

One trap that catches people: recording rules can double the problem instead of relieving it. A rule written to reduce query load, but which preserves the high-cardinality label, runs every evaluation interval and writes a fresh series for every combination. You now have the original explosion and a derived copy of it. If prometheus_rule_evaluation_duration_seconds is spiking, check your rules before anything else.

If the instance is OOM-killing repeatedly and you cannot isolate the source, running a second short-retention Prometheus as a diagnostic scraper lets you narrow it down without gambling with the one that is still notionally working.

Preventing it

Drop what you do not need at scrape time rather than filtering at query time. Relabel configs handle this, and the label never enters the index at all:

yaml

metric_relabel_configs:
  - regex: "user_id|session_token|request_id"
    action: labeldrop

For latency and size distributions, native histograms are worth looking at seriously. Classic histograms create a series per bucket per label combination, which is where a surprising amount of cardinality quietly lives. Native histograms hold the whole distribution in a single series with dynamic buckets, and they have matured considerably in the Prometheus 3.x line.

The habit that matters most is cheaper than any of this. Before adding a label, ask how many distinct values it can take in production, not in staging. If the answer is "it depends on the user" or "one per request", that value belongs in a log line or a trace, where the storage model is built for it. Metrics are for things you count. Logs and traces are for things you identify.

One footnote on security

Because this is rarely mentioned: the Prometheus Pushgateway ships with no authentication. Any process that can reach it over the network can push metrics with arbitrary label values.

Systems Hardening lays out the consequence: a compromised pod in a cluster without network policies can deliberately inject high-cardinality labels until Prometheus exhausts memory and dies. Roughly a gigabyte of memory per hundred thousand injected series. It is a cheap way to blind the people investigating you, and it does not look like an attack while it is happening. It looks like a monitoring problem.

Worth a network policy, if you have a Pushgateway exposed inside your cluster.

Author note

I am Mohan Gopi, an Associate DevOps Engineer at Frigga Cloud Labs, working across AWS, GCP, and Azure with Prometheus, Loki and Grafana as our observability stack. I wrote this because cardinality is the rare failure where the person creating the risk and the person paying for it are almost never the same person. The pattern I keep seeing is teams treating a rising memory graph as a capacity question and buying a bigger instance, which works for about a month, or discovering the whole thing mid-incident when the dashboards go blank. Counted before it is merged, a label is a five-second question about how many values it can take. Counted after, it is a monitoring outage that arrives on the worst possible day and takes your evidence with it.

Let us connect on LinkedIn → Mohan Gopi

One Prometheus Label Can Take Down Your Monitoring