Watch the first two minutes of any investigation and you will notice that almost none of it is investigation.

An alert fires for a service. Before anybody can reason about what went wrong, somebody has to establish what the service even is. Which cluster it runs on. What database it talks to. Which dashboard covers it. Whether it was deployed recently, and by which pipeline. Who owns it.

None of those are questions about the incident. They are questions about the system, and the answers were true yesterday and will be true tomorrow. But they get asked fresh, every time, because nowhere in the toolchain is that picture kept.

That is the part I think we have been quietly accepting as normal.

Two ways to arrive at an answer

There is a real architectural fork here, and it decides how fast anything downstream can be.

The first approach assembles context when it needs it. An alert arrives, and the system goes and works out what is related to it. Query the metrics, look up recent deploys, find the logs, infer the dependencies. Then reason over what came back.

This works. It is also doing discovery and diagnosis in the same breath, under time pressure, every single time. And it starts from close to nothing, so the quality of the answer depends heavily on whether the assembly step happened to fetch the right things.

The second approach keeps the picture. The relationships between services, infrastructure, pipelines, dashboards and owners are maintained continuously, so when an alert arrives the topology is already known. The only new work is the diagnosis.

The difference is not speed for its own sake. It is that the second approach never has to guess at what is relevant, because relevance was established before anybody was under pressure.

What a maintained picture actually contains

It is worth being concrete about this, because "context" gets used so loosely that it stops meaning anything.

For a single service, the useful picture is something like: which compute it runs on, what sits in front of it, which datastores and caches it depends on, which monitoring covers it, where its errors land, which pipeline deploys it, which team owns it, how critical it is, and which environment this instance is.

Every one of those is a fact about the system rather than a fact about today. They change occasionally, through deliberate acts like a migration or a reorganisation. They do not change during an incident.

So there is no good reason to rediscover them during one.

Why this changes the diagnosis, not just the lookup

Here is the part that took me a while to see properly, and it is more interesting than the time saving.

Suppose latency on a service crosses its threshold. A system that knows only the alert has to work out where to look. A system that already knows this service sits behind a load balancer, talks to a relational database and a cache, and was deployed forty minutes ago by a specific pipeline, can go straight to checking those specific things.

It is the difference between searching and checking. Searching is open-ended and you stop when you find something plausible. Checking is bounded, and you can say what you looked at and what came back clean.

That second property is the one that matters during an incident. An answer that names what was ruled out is worth considerably more than an answer that names only what was found, because the ruled-out list is what stops three people re-investigating the same thing.

Ownership is a technical field

One part of that picture gets treated as administrative overhead and I want to argue against that.

Ownership, criticality and environment sound like metadata for a spreadsheet. In practice they are what turns a diagnosis into an action.

Knowing the cause is not the end of an incident. Somebody has to do something, and the gap between "the payments database is saturated" and "the payments team has been paged" is where a surprising number of minutes disappear. If ownership lives in a wiki that is eight months out of date, you have a correct diagnosis and no route to a fix.

Criticality does similar work. The same technical symptom on a critical production service and on a staging instance are not the same event, and a system that cannot tell them apart will treat them identically. Which means either it wakes people for staging, or it does not wake them for production. Both are bad, and neither is a diagnosis problem.

This is why I now think of the graph as part of the investigation rather than as documentation about it.

The honest constraint

A maintained picture has a real cost, and it is not the storage.

It has to stay accurate. A topology that is subtly stale is worse than no topology, because it will confidently tell you a service depends on something it no longer does, and you will spend twenty minutes on that. Anything of this shape lives or dies on whether it is derived from the systems themselves rather than declared by hand.

Which means the discipline is not in building the graph. It is in never letting a human be responsible for updating it.

This is roughly how we have built Vörr. It reads from the code, the cloud accounts, the pipelines and the monitoring, and maintains the relationships between them rather than reconstructing them per question. When something goes wrong, the questions it asks are narrowed by what it already knows about that service, and the answers come back with the source attached.

I am not claiming that is a novel idea. It is the same instinct that makes a good architect fast, and all we have done is stop expecting that picture to live in one person's head.

What I would check on your own setup

Time the discovery phase separately from the diagnosis phase during your next incident. Not the whole thing, just the part before anybody forms a hypothesis. Most teams have never measured this and are surprised by it.

Ask where your dependency information lives. If the honest answer is that it lives in the people who have been there longest, that is fine until it is not, and the day it stops being fine is not a day you choose.

Then check whether your ownership data is derived or declared. Declared data is accurate on the day it is written. Derived data is accurate on the day you read it, which is the only day that ever matters.

The alert is not the beginning of the investigation. It is the point at which you find out how much of the investigation you already did.

Author note

I have started thinking about this as the difference between a map and a search party.

Both will eventually get you to the same place. Only one of them is useful in the ten minutes when it matters, and building the map is work you can do on a quiet afternoon rather than at two in the morning with three people watching.

Mohan Gopi, Associate DevOps Engineer at Frigga Cloud Labs, working across AWS, GCP and Azure. LinkedIn if you want to disagree with any of this.

The Investigation Starts Before the Alert Does