From incident to fix

Use caseIntermediate15 min

An incident asks the hardest questions at the worst time: is this real, what is affected, who owns it, what changed, and is it over yet. Run the incident itself in your incident tool, Shankh if you use it, and make the mitigation decisions yourself. Vörr gives the responder answers from your connected cloud, observability and CI/CD tools, each with its source, and CodeZero turns the permanent fix into a plan someone approves before anything changes.

When you'd do this

You have been paged, several alerts are firing at once, and you need to know within minutes whether this is one incident, several, or noise.
A shared dependency such as a database, queue or cache is degraded or about to go into maintenance, and you need to know everything that will break.
You are the incident commander and have to name an owner, a likely cause and a mitigation while people are waiting on you.
Stakeholders want an update every 30 minutes and each one has to be accurate, not optimistic.
The mitigation has gone out and you need evidence that the service has recovered before you stand people down.
The incident is over and you need a permanent fix and a postmortem that does not depend on someone's memory of the call.

This page covers the incident as a whole. The detailed debugging operations (symptom per source, top errors, logs, traces, code path and configuration) are on Debug a production error, and this page links there rather than repeating them. If the trigger is a pipeline that failed, start with Investigate deployments and CI failures.

Before you start

Vörr: your organisation set up, with the observability, alerting, error-tracking and CI/CD tools you rely on connected. Every source you have not connected is a question that comes back as no connector during the incident, so connect them before you need them.
Vörr: cloud accounts connected with read-only credentials and scanned, if you want blast radius and dependency answers for databases, queues and other cloud resources.
Vörr: the connector added to Claude, ChatGPT or another MCP assistant by each responder, with their own consent. Answers draw on the organisation's connected sources, and workspaces label what comes back. ChatGPT needs a paid plan with Developer mode turned on.
Vörr: critical services mapped in the service map by an administrator, so orders, orders-api and orders-prod resolve to one service under pressure.
CodeZero: installed in VS Code with Claude Code, Codex, GitHub Copilot or Cursor as the agent, and the organisation's repositories indexed, for the code side of blast radius and for the permanent fix.
Your incident tool: where the incident is declared, acknowledged, escalated, updated and resolved. Neither Vörr nor CodeZero does this.

If any of this isn't set up yet, start here:

Who does what during an incident

Your incident tool (for example Shankh): Opens incidents from alerts, pages people, tracks acknowledgement, severity and escalation, holds progress updates and the stakeholder page, and records the resolution.
You and your team: Decide severity, choose and run the mitigation (rollback, scale, feature flag, failover) in your own tools, and approve the permanent fix.
Vörr, from Claude or ChatGPT: Reads the running system: alerts, silences, coverage, metrics, errors, deploys, dependencies, owners and root-cause scoring, each answer labelled with its source and environment. It is read-only towards your systems from the assistant.
CodeZero, in VS Code: Reads code across indexed repositories, including callers and the tables code reads and writes, and carries the permanent fix as a plan with named repositories and verification commands.

The two products are separate. You carry findings between them, usually by pasting the service, deploy, repository and file Vörr identified into a CodeZero prompt.

Operation 1: Triage what is firing and whether it is real

Vörr lists the alerts firing now across connected alerting sources, with the rule, its threshold and its recent history, and, where Grafana alerting is connected, the routing or contact point each one goes to.
Ask for silences in the same window. A silenced alert on the affected service is a reason the page you got may understate the problem.
Ask Vörr's coverage view which sources can answer for the affected service and why an answer could be wrong: silences, rules that are failing to evaluate, missing connectors. It also reports services nobody watches.
Group the firing alerts by service and start time. Alerts on several services that started within the same minute usually share a cause.

What is firing

Using Vörr, list every alert firing in production right now. For each: rule, service, threshold, when it started firing, how often it has fired in the last 7 days, and where it routes. Group them by service and start time.

Is it real

Using Vörr, for orders in production: are any alerts silenced, are any rules failing to evaluate, and which connected sources can actually answer health questions for it? Tell me anything that could make a quiet signal misleading.

A good triage answer separates alerts with a long history of flapping from ones that have never fired before, names the source of each, and says where coverage is missing. If an alert has fired every night this week at the same time, say so in the incident channel before anyone starts a rollback.

Operation 2: Pin the environment and the scope across sources

Ask Vörr for the affected service's error rate, latency and throughput per connected source, with the matched service name, match quality and environment on each figure. Vörr never averages sources together.
Confirm the environment. A staging alert routed to the production channel is a common false start, and every Vörr result carries the environment it came from.
Ask which other services show a change in the same window, so you know whether this is one service or a wider event.
For the detailed version of this check, including comparison windows and disagreement between sources, follow Operation 1 on Debug a production error.

Scope

Using Vörr, is orders degraded in production right now? Give error rate, p99 and throughput per connected source, with the matched name, match quality and environment on each. Which other production services changed in the same 15 minutes?

Check the match quality. A name or unmapped match can be the right service, but re-ask with the explicit name from the tool before you put the figure in a status update.

Operation 3: Size the blast radius

For a cloud resource such as a database, cache or queue, Vörr returns the precomputed upstream and downstream blast radius with true counts, so you see how many resources are affected and which ones. The list behind the counts may be truncated, and the answer says so when it is. The same relationships are in the web app's Architecture view.
For a service, Vörr returns its service-graph callers and callees from connected tracing and APM data: what is calling it in production today.
In CodeZero, ask which repositories call the affected service's routes, reference it through Kubernetes and Helm config, produce to or consume from its queues, or read and write its tables. These cross-repository edges are inferred, show file:line evidence (HTTP and config edges also show a confidence), and can be checked with Verify.
Put the runtime list and the code list side by side. The runtime list shows what is calling now; the code list shows what could call, including nightly jobs and rare paths that have not run yet today.

Resource blast radius (Vörr)

Using Vörr, if the orders database in production is unavailable, what breaks? Give the upstream and downstream blast radius with true counts, list each affected resource and service with its source and confidence, and include anything that reads a replica.

Runtime callers (Vörr)

Using Vörr, which services call orders in production according to the service graph, and which does orders call? Include call volume where the source has it.

Code dependents (CodeZero)

Which repositories call the orders service's routes, reference ORDERS_DB_URL or the orders replica in config, or read or write the orders tables? Show the file:line evidence for each edge, and the confidence for HTTP and config edges.

A good blast-radius answer has counts, sources and confidence on every item, and says what it could not see: accounts not connected, resource types not indexed, or a scan older than the change you are worried about.

Why blast radius gets underestimated

A search of one repository finds its direct dependents at best, while the services that break are often transitive dependents that reach it over HTTP, a queue or a replica. Infrastructure relationships such as a connection string set in another repository's Helm values, or a resource wired up only in the cloud account, are not references in the code, so a literal search cannot find them and cannot tell a true zero from a search that never looked.

Explore cloud infrastructure

Operation 4: Find who owns it

Vörr's resource detail gives the workspace and environment a resource or service is mapped to, along with its tags and any mapping gaps. An unmapped resource is itself a finding: nobody has claimed it.
Vörr or CodeZero can list the repository's owners by commit history, and the repositories mapped to the service with their branch-to-environment mapping.
Page the owning team through your incident tool. Commit history shows who knows the code, which is not always who is on call.

Owner

Using Vörr, which workspace and environment are the orders database and the orders service mapped to, which repositories deploy to orders in production, and who are the most frequent recent committers to each repository?

Check that the workspace mapping matches your on-call rota. If the resource has no workspace, record it as a follow-up for the postmortem.

Operation 5: Build the timeline

Ask Vörr's deploy feed for every deploy in the window before the first alert, in any service, and for the deploy that preceded the symptom on the affected service and its dependencies.
Add config commits from CodeZero's recent changes on the repositories that hold Kubernetes, Helm or Terraform config, then combine deploys, config commits, first alert and first error into one ordered list for the incident record.

Deploy timeline (Vörr)

Using Vörr, the first orders alert fired at 02:14. List every production deploy anywhere in the organisation between 01:00 and 02:14, with service, source, time, the commit or revision where the source reports one, and any regression window Vörr detects around them (with Grafana connected).

A deploy that landed after the first alert is not the cause. Full detail: Investigate deployments and CI failures.

Operation 6: Score the likely root causes

Vörr's root-cause scoring rates candidates from 0 to 100 across memory, CPU, disk, network, deployment, dependency, database and external, and names the preceding deploy.
It also reports recurring patterns across past incidents on the same service. A pattern that matches is the fastest route to a known mitigation.
If you need to go from a score to a trace, a log line or a file, switch to the operations on Debug a production error and come back with the result.

Root-cause scoring

Using Vörr, run root-cause scoring for orders in production since 02:00. Show the score per category, the preceding deploy, and any recurring pattern from past incidents, with what each pattern matched on.

Scores rank where to look; they do not prove a cause. Say so in the incident channel when you share them, and pair a high score with a supporting deploy, trace or metric before you act on it.

Operation 7: Decide on a mitigation

Neither product executes mitigations. Vörr is read-only from the assistant, and CodeZero changes code in a working tree, not production. What they do is give you the facts each option depends on, so the incident commander makes the decision with evidence.

Rollback check (Vörr)Using Vörr, what was the 01:52 orders deploy (source, revision, environment) and which services call orders?

Rollback check (CodeZero)What changed in that commit range, and did it include a migration?

Scale check

Using Vörr, show CPU, memory, restarts and pods ready for orders and its direct dependencies over the last 2 hours. Is orders resource-bound, or is it waiting on something it calls?

Flag check (CodeZero)

Where is the orders_async_writes flag read across all indexed repositories, and what code paths does turning it off change?

Mitigation	Facts to check first	Where they come from
Roll back a deploy	Which deploy preceded the symptom, what else shipped in the same release, whether a migration ran, who else calls the changed route	Vörr deploy feed and root-cause scoring; CodeZero recent changes and callers of the route
Scale out	Whether the service is resource-bound: CPU, memory, restarts, pods ready, top consumers by pod or deployment; whether the bottleneck is actually a dependency	Vörr service health and Kubernetes top consumers; Vörr service graph
Turn off a feature flag	Where the flag is checked and which code paths it guards, across repositories	CodeZero search and callers across indexed repositories
Fail over	What depends on the primary and on the replica, how many dependents each has, and which are in other accounts or regions	Vörr blast radius with true counts and Architecture; CodeZero data-layer edges

Record the decision, the evidence and who made it in the incident tool. Then run the mitigation in your own deployment, scaling or flag tooling.

Operation 8: Draft status updates

Ask the assistant to draft an update from what Vörr has returned so far: current impact, affected services with counts, what is being done, and when the next update comes. Ask it to cite the source for each fact.
Remove anything that is not backed by a source, and anything inferred that you have not confirmed.
Post it in your incident tool. In Shankh, add it as a progress update, not an internal comment: internal comments never appear on the stakeholder live page, and progress updates do.

Stakeholder update

Using only what Vörr has returned in this conversation, draft a stakeholder update in plain language: current impact, affected customer-facing services, what we are doing, and next update at 03:30. List the source behind each statement separately so I can check it. Do not speculate about cause.

The assistant drafts; a person posts. Assume anything posted to a stakeholder page may be read widely, and keep internal detail such as hostnames and commit messages out of it.

Manage an incident in Shankh

Operation 9: Confirm recovery

After the mitigation, ask Vörr for a comparison window: the period after the mitigation against the period before the incident and the same window seven days ago, per source.
Check the current SLO status of the affected services, with remaining budget where the source reports it (for example Dynatrace).
Ask for the alerts that fired during the incident and confirm they have cleared, and that no silence added during the incident is still in place.
Check the dependents from Operation 3, not just the service that paged. A downstream job that failed during the window may need rerunning.

Recovery

Using Vörr, the rollback finished at 03:05. Compare orders, checkout and invoicing error rate and p99 from 03:10 to 03:40 with 01:00 to 02:00 and with the same window 7 days ago, per source. Are the incident's alerts all cleared, and what is each SLO's current status, with remaining budget where the source reports it?

Resolve the incident in your incident tool only when every source agrees and the alerts have cleared. In Shankh, resolution records a resolution note; MTTA runs to acknowledgement and MTTR to resolution.

Operation 10: Plan the permanent fix

Take the evidence into CodeZero: the deploy, the commit range, the file, the dependents and the mitigation you used.
Use Planned mode, or /plan, so the agent makes no code changes until a plan is approved. The plan names the repositories it touches, its phases and its verification commands, and the agent cannot approve its own plan.
Ask for a reproduction first, a failing test or equivalent check, and for every dependent from Operation 3 to be accounted for.
Review the plan with the owners from Operation 4. After approval, edits in the plan's repositories go ahead, an edit in another checkout of an organisation repository asks first, paths outside every checkout are refused, and only the declared verification commands run without asking. A fully clarified plan can be handed to Coffee Mode, which never commits, pushes or switches branches.

Plan the fix

/plan The 01:52 orders deploy made the order-write path hold a row lock across an external call; orders timed out from 02:14 and checkout and invoicing failed with it. We rolled back at 03:05. Reproduce the lock contention with a test first, then fix the write path. Include the test suite and the new test as verification commands, and list every repository that calls the orders write route or reads the orders tables.

Before approval

Before I approve: which callers of the orders write route are not covered by tests in this plan, and does anything else hold a lock on the orders table?

A good plan has a reproduction that fails before and passes after, names every affected repository, and answers the clarification questions the agent raises rather than assuming. The full flow is on Debug a production error, Operation 10.

Operation 11: Gather postmortem inputs

The timeline from Operation 5, extended with the mitigation, recovery and resolution times.
The evidence behind each decision, with its source: alerts, metrics per source, root-cause scores, blast radius counts.
Recurring patterns Vörr's root-cause scoring found across past incidents on the same services. If this is the third time, that is the headline of the postmortem.
Coverage gaps found during the incident: silenced alerts, rules that did not fire, unwatched services, unmapped resources, sources that were not connected.
The CodeZero plan for the permanent fix, its verification and its status.
In Shankh, the postmortem draft is assembled from the incident record (timeline, severity changes and progress updates), with fields for what happened, investigation notes, contributing factors and the AI postmortem cause. Fill in what the record does not show yourself.

Postmortem inputs

Using Vörr, for the orders incident from 02:14 to 03:05: give the deploys, alerts and recovery in one ordered timeline with sources, any recurring pattern from past incidents on orders, checkout or invoicing, and every coverage gap you found, such as silences, failing rules or services with no connected source.

Keep each item tied to its source. A postmortem that says p99 reached 4 s should say which tool measured it.

Without CodeZero or Vörr

An experienced responder can do a great deal with Claude and their existing tooling. Claude Code with kubectl, a cloud CLI and gh installed and credentialed, plus vendor MCP servers for Datadog, Grafana or Sentry, can list firing monitors, read pod status, tail logs, list workflow runs and search the repositories you have cloned. ChatGPT or Claude chat can reason well over pasted alerts and graphs and draft a clear update. Under time pressure, the cost is the setup and the stitching.

Operation	Claude Code / ChatGPT on their own	With CodeZero and Vörr
Triage alerts	Possible per alerting tool with its MCP server or CLI. Silences and failing rules need separate queries; no view of what is unwatched	Vörr lists firing alerts, history, silences and, for Grafana alerting, routing, and reports coverage and unwatched services
Scope and environment	One tool at a time, each with its own name for the service	Vörr resolves the name and labels every figure with source, match quality and environment
Blast radius	grep in cloned repositories, or cloud CLI queries per account. HTTP, queue and replica links are not source references, so the list is short and looks complete	Vörr precomputed dependents with true counts and service-graph callers; CodeZero cross-repository and data-layer edges with evidence
Owner	git log in cloned repositories; workspace ownership lives elsewhere	Vörr workspace and environment mapping; owners by commit history across indexed repositories
Timeline	gh for GitHub Actions plus separate access to Argo CD, Jenkins and the rest, merged by hand	Vörr organisation-wide deploy feed and preceding deploy
Root cause	The agent reasons over what it managed to gather	Vörr root-cause scoring across eight categories with recurring patterns from past incidents
Status updates	Good drafts from pasted facts; sources are whatever you pasted	Drafts from Vörr answers that each carry a source
Permanent fix	Edits directly; no plan approval or write-scope guard	CodeZero plan with named repositories, approval, scoped edits and declared verification

The real costs: every tool needs its own MCP server or CLI and credentials on the responder's laptop, and the on-call engineer at 2 a.m. may not have them all; the service has a different name in each tool; building one timeline means querying every CI system and lining up the clocks; and an empty result looks the same whether the service is fine, the name was wrong or the tool was never connected. Broad searches also cost context and steps, so an agent tends to stop at the first plausible answer, which in an incident is how a blast radius gets understated.

With CodeZero and Vörr

Fewer tools to hold (Vörr): Alerts, metrics, errors, deploys and dependencies from more than 30 connected tools through one connector, with no observability credentials on the laptop. Stored credentials are never exposed to the assistant.
The right service, labelled (Vörr): Names resolved through the service map and aliases, with source, match quality and environment on every result.
An honest zero (Vörr): Coverage says which sources can answer, flags silences and failing rules, and a missing capability returns no connector instead of nothing.
Blast radius with counts (Vörr and CodeZero): Vörr's precomputed dependents and service graph for the running system; CodeZero's cross-repository and data-layer edges for the code, each with evidence.
One timeline (Vörr): Deploys across connected CI/CD sources in one feed, the preceding deploy and recurring patterns across past incidents.
A fix someone approves (CodeZero): A plan with repositories, phases and verification commands that the agent cannot approve itself, and confirmation before any edit outside the plan's repositories.

Neither product replaces the incident tool or the people deciding. They shorten the time between a question and a sourced answer, and they make the permanent fix go through the same review as planned work.

Illustrative example

Worked example: orders database lock contention

At 02:14 the on-call engineer is paged for orders. Within two minutes, alerts are also firing for checkout and invoicing. Shankh has the incident open and acknowledged; the engineer opens Claude with the Vörr connector.

You: Using Vörr, what is firing in production, is any of it silenced, and can we trust it?

Vörr:
  Firing (grouped by start)
    02:14  orders     error rate > 5%          Prometheus    first fire in 30 days
    02:14  orders     p99 > 1 s                Datadog APM   first fire in 30 days
    02:16  checkout   error rate > 2%          Datadog APM   fired twice in 7 days
    02:17  invoicing  job failures > 0         Alertmanager  first fire in 30 days
  Silenced
    orders-db  connections > 80%   silenced until 06:00 (maintenance window)
  Coverage
    orders, checkout: metrics, errors, traces, deploys
    reporting-nightly: no connected source (unwatched)

Illustrative example

The silenced database alert matters: it would have fired. The engineer asks for the blast radius before anyone touches the database.

You: Using Vörr, what depends on the orders database in production? Counts and sources.

Vörr:
  orders-db (RDS, prod, workspace: commerce)
    downstream dependents: 7 (true count)
      orders-api          service graph + scan   explicit
      orders-db-replica   scan                   explicit
        reporting-nightly   reads replica (scan)  inferred
      checkout            service graph (calls orders-api)
      invoicing           service graph (calls orders-api)
      ... 2 more (fulfilment worker, admin-console)

You (CodeZero): Which repositories read the orders tables or reference the orders replica?

CodeZero:
  reporting   jobs/nightly_orders.py:31      reads orders, order_lines
  platform    helm/reporting/values-prod.yaml:12  ORDERS_REPLICA_URL     high
  admin       src/orders/export.ts:18        reads orders

Illustrative example

Next the timeline and the scores.

You: Using Vörr, deploys before 02:14 anywhere in prod, and root-cause scoring for orders.

Vörr:
  Deploys 01:00-02:14 (prod)
    01:52  orders-api  Argo CD         3be10a2..7c4d9e0
    00:40  web         GitHub Actions  copy changes
  Root-cause scoring (orders, since 02:00)
    database 88  deployment 79  dependency 52  network 11  cpu 9  memory 6
    preceding deploy: orders-api 01:52
    recurring pattern: similar database-lock signature on orders, 2026-08-14

Illustrative example

Worked example: decision, updates, recovery and fix

Owner: Vörr maps orders-api and orders-db to the commerce workspace; the most frequent recent committer to the orders repository is on the commerce rota and is paged through Shankh.
Decision: the incident commander chooses to roll back the 01:52 deploy. CodeZero's recent changes show the release changed the order-write transaction and ran no migration, so a rollback is safe. Failing over to the replica was rejected because the reporting job and admin export read it. The rollback is run in Argo CD at 02:58.
Update: Claude drafts a progress update from the Vörr answers, citing each source. The engineer removes the internal hostnames and posts it in Shankh as a progress update.
Recovery: from 03:10 to 03:40 Vörr shows orders, checkout and invoicing back to baseline in Prometheus and Datadog, all four alerts cleared, and the orders SLO status back within target. The database silence is still in place until 06:00 and is noted. The invoicing job is rerun.
Permanent fix: CodeZero plans a change in the orders repository only, with a test that reproduces lock contention on the write path, and lists the reporting and admin readers as unaffected. The owner approves it; the PR is opened by a person.
Postmortem inputs: the ordered timeline with sources, the recurring pattern from August, the silenced alert, reporting-nightly being unwatched, and the plan.

Follow-ups: connect a source for reporting-nightly, review whether maintenance silences should cover connection alerts, and confirm the admin export's read of the orders tables (src/orders/export.ts:18) in Graph.

Verify and follow up

Provenance: every figure in an update or postmortem should name its source and environment. Re-ask with an explicit name when Vörr reports a name or unmapped match.
Coverage: treat a quiet service as healthy only when Vörr's coverage says a source can answer for it and nothing is silenced.
Blast radius: check connected accounts and the scan time before calling a dependents list complete, and Verify inferred CodeZero edges you are relying on for a decision.
Timeline: line up clocks across sources and check the environment on each deploy.
Recovery: repeat the comparison window in every source and check the dependents, not just the service that paged.
Fix: the reproduction should fail before the change and pass after. Run the declared verification commands yourself if they ran unattended.

Next operations: dig into the cause with Debug a production error, close coverage gaps with Monitor service health, and review exposure or infrastructure findings with Review security and exposure.

Limits

Neither product runs the incident. Declaring, paging, escalation, acknowledgement, stakeholder pages and resolution stay in your incident tool. Vörr and CodeZero have no PagerDuty or Opsgenie integrations; Shankh accepts PagerDuty webhooks as an alert source, and has no Opsgenie integration.
Neither product executes mitigations. Vörr is read-only towards your systems from the assistant: it cannot roll back, scale, flip a flag, fail over, restart or silence anything.
Vörr only answers from tools and accounts you have connected. Blast radius is bounded by connected accounts and the latest scan; a resource created since the scan may be missing.
Name resolution is only as good as the service map. Name, suggested and unmapped matches are labelled because they can be wrong.
Root-cause scoring and recurring patterns rank candidates; they do not prove a cause.
Status updates drafted by the assistant are drafts. Shankh attaches a Vörr AI analysis to new critical and high-severity incidents when the organisation has configured an LLM key, and its postmortem draft is assembled from the incident record.
CodeZero indexes committed code on the default branch, its references are matched by name, and its callers and cross-repository edges are heuristic, not compiler-accurate.
Check before an incident, not during one, that every on-call engineer has added the Vörr connector to their assistant.

When you'd do this

Before you start

Who does what during an incident

Operation 1: Triage what is firing and whether it is real

Operation 2: Pin the environment and the scope across sources

Operation 3: Size the blast radius

Why blast radius gets underestimated

Operation 4: Find who owns it

Operation 5: Build the timeline

Operation 6: Score the likely root causes

Operation 7: Decide on a mitigation

Operation 8: Draft status updates

Operation 9: Confirm recovery

Operation 10: Plan the permanent fix

Operation 11: Gather postmortem inputs

Without CodeZero or Vörr

With CodeZero and Vörr

Worked example: orders database lock contention

Worked example: decision, updates, recovery and fix

Verify and follow up

Limits

Related