Debug a production error

When you'd do this

• An error rate or latency percentile has moved and you need to know whether it is real, where it comes from and since when.
• A new exception is at the top of the error tracker and you need the code path that throws it.
• Someone reports that checkout is slow and the local git log shows nothing that obviously explains it.
• Two dashboards disagree about how bad it is, and you need to know which one to believe.
• You have a likely cause and want to check what else depends on the failing service before you change anything.
• A fix has shipped and you need evidence that the symptom has actually gone.

If an incident is open and people need coordinating, start with From incident to fix. If the problem is a build or deploy pipeline that failed, use Investigate deployments and CI failures. For ongoing checks of a service that is not currently broken, see Monitor service health.

Before you start

• Vörr: your organisation set up, repositories connected, and the observability, error-tracking and CI/CD tools you rely on connected. The more sources are connected, the more of the questions below have an answer rather than a coverage gap.
• Vörr: the connector added to Claude, ChatGPT or another MCP assistant, with your own consent. You see the workspaces you are a member of. ChatGPT needs a paid plan with Developer mode turned on.
• Vörr: important services mapped in the service map by an administrator, so checkout, checkout-svc and web-checkout resolve to one service. Unmapped services still work, but matches are labelled name or unmapped.
• Vörr: cloud accounts connected with read-only credentials if you want dependency and blast-radius answers for cloud resources.
• CodeZero: installed in VS Code with Claude Code, Codex, GitHub Copilot or Cursor as the agent, and your organisation's repositories indexed. You do not need every repository cloned.

If any of this isn't set up yet, start here:

• Connect observability and CI tools
• Use the Vörr MCP connector
• Install and connect CodeZero

How the work splits

Vörr answers questions about the running system from your assistant: metrics, errors, logs, traces, deploys, alerts, dependencies and coverage, each with its source. CodeZero answers questions about the code and carries the change: search across repositories, history, blame, configuration the code reads, and a plan with verification commands. They are separate products. You move the findings from one to the other, usually by pasting the repository, file and deploy that Vörr identified into a CodeZero prompt.

Operation 1: Confirm the symptom in every source

1. Vörr resolves the service name you use through the service map, confirmed aliases and catalogue names before trying the raw name. If the name is ambiguous, ask it to list the real service, job and project names in each connected tool.
2. Vörr queries every connected source that can answer and returns error rate, p95 latency (plus p50 and p99 where the source reports them) and throughput per source. It never averages them into one number.
3. Ask for a comparison window, for example the last hour against the same hour seven days ago. Vörr reports the delta per source and flags when sources disagree. CloudWatch metrics are also checked against a 14-day baseline and labelled normal, elevated or unprecedented, or unknown where there is no baseline yet.
4. Check coverage: ask which sources can answer for this service and whether anything could make the answer wrong, such as a silence or a failing alert rule.

A good result names the source, environment and match quality next to every figure. If one tool says 4% and another 0.3%, that disagreement is a finding in itself, often a difference in how retries or status codes are counted. A capability no connected tool provides comes back as no connector, not as zero. Coverage and unwatched services are covered in more depth in Monitor service health.

Operation 2: Pull the top errors

1. Vörr returns the top error issues for the service from the connected error tracker or APM: counts, culprit, a redacted sample and the stack.
2. Ask for the window the symptom started in, so a long-running background error is not mistaken for the new one.
3. Note the culprit frame and the first in-house frame in the stack. You will take those into the code in Operation 8.

Check that the sample is redacted and that the last-seen time shows the error is still occurring. Compare the count with an earlier window: an error that has been firing for weeks at the same rate is rarely the cause of today's change.

Operation 3: Read and filter logs

1. Vörr reads log lines from connected log sources (for example Loki, Elasticsearch or Splunk), redacted, along with log volume over time.
2. Filter by level, by a message fragment from the error sample, or by request or trace ID. You can also ask for a raw PromQL query against Prometheus-compatible sources, or a LogQL query through a Grafana Loki datasource.
3. Compare log volume before and after the symptom started. A sudden drop is as informative as a spike.

Good log evidence carries timestamps and the source it came from. If the log source for this service is not connected, Vörr says so rather than returning an empty list.

Operation 4: Read traces and find the bottleneck span

1. Vörr summarises traces for the service from connected tracing tools (for example Tempo, Jaeger, Zipkin or an APM) and lists slow or failing traces.
2. Open a single trace as a waterfall. Vörr identifies the bottleneck span, the one where the time actually goes.
3. Ask for p99 per operation to see whether one downstream call explains the overall latency.

If the bottleneck span belongs to another service, the investigation moves there. This is the point where a local-only investigation usually goes wrong.

Operation 5: Correlate with deploys across the organisation

1. Ask Vörr's deploy feed which deploys, in any service and any connected CI/CD source, happened in the window before the symptom started. With Grafana connected, Vörr also shows the regression windows it detects alongside those deploys.
2. For the candidate deploy, note the repository and the commit or revision where the source reports one. CodeZero will read it in Operation 8.

The deploy that matters is often not in the repository you have open. Full detail: Investigate deployments and CI failures.

Operation 6: Score the likely root causes

1. Vörr's root-cause scoring rates candidate causes from 0 to 100 across memory, CPU, disk, network, deployment, dependency, database and external factors.
2. It reports the preceding deploy and any recurring pattern from past incidents on the same service.
3. For a Kubernetes workload, also ask for infrastructure health (restarts, pods ready, CPU and memory) and the top consumers by pod or deployment, to rule resource pressure in or out.

Treat the scores as a ranking of where to look, not a verdict. A high dependency score plus a bottleneck span in another service plus a deploy to that service is a strong case. A high score with no supporting trace or deploy is a lead to check.

Operation 7: Check dependencies, callers and blast radius

1. Vörr returns the service-graph callers and callees from connected tracing or APM data, so you can see who calls the failing service and what it calls.
2. For cloud resources such as a database or queue, Vörr gives the precomputed upstream and downstream blast radius with true counts.
3. In CodeZero, use Graph or ask Chat which repositories call the affected route or read the affected table. These cross-repository edges are inferred from HTTP calls matched to routes, deployment config, queue producers and consumers, and the tables each repository reads and writes. Each edge shows file:line evidence, and HTTP and config edges also show a confidence; Verify checks the call site.

Runtime callers and code callers answer different questions. The service graph shows what is calling today. CodeZero's Graph shows what could call, including paths that only run under rare conditions. Use both before deciding on a mitigation such as a rollback.

Operation 8: Find the code path

1. Ask Vörr for an error-to-code investigation: from the culprit and stack it finds the repository and file that define the behaviour, using the same code index.
2. In CodeZero, open that file even if the repository is not cloned. Indexed files come with their last-commit provenance.
3. Use CodeZero's recent changes for the repository or path, then Blame and file history in Git Tree, to find the commit that introduced the behaviour and compare it with the deploy from Operation 5.
4. Ask CodeZero Chat for the callers and callees of the function involved, across repositories.

CodeZero's references are matched by name and callers are heuristic, so same-named symbols can appear. Confirm the important ones by reading the call site.

Operation 9: Check the configuration the code reads

1. Ask Vörr for a config-to-code investigation, or use CodeZero to find where each setting is defined in code, in .env.example and in deployment config.
2. Find where each setting is actually set: Kubernetes manifests, Helm values or Compose files in another repository are often the real source, not the default in code.
3. Compare the value in each environment with the code default before deciding whether the fix is a code change or a config change.

A fix to a code default does nothing if production sets the value elsewhere. Make sure the plan in Operation 10 changes the place production reads.

Operation 10: Reproduce and plan the fix

1. In CodeZero, describe the cause with the evidence you gathered: the deploy, the commit, the file and the trace. Use Planned mode if you want no code changes until you approve a plan.
2. Ask the agent to reproduce the problem first, for example with a failing test or a query plan, and to put that reproduction in the plan.
3. The plan names the repositories it touches, its phases and the verification commands. If a repository is not checked out, CodeZero offers to clone it. Edits in the plan's repositories go ahead, an edit in another checkout of an organisation repository asks first, paths outside every checkout are refused, and the agent cannot approve its own plan.
4. After approval, outside Coffee Mode, only the declared verification commands run without asking. If the plan is fully clarified, you can hand the remainder to Coffee Mode. It refuses commits, pushes and branch switches unless the approved plan declared them, so you review the result afterwards.

A good plan has a reproduction step that fails before the change and passes after it, verification commands you can run yourself, and an explicit list of affected callers. If the agent asks a clarification question, answer it. That is the guard against a silent assumption.

Operation 11: Confirm the fix after deploy

1. Once the fix is deployed, compare the symptom per source over a window after the deploy with the window before the regression, and check that the top error has stopped growing and the bottleneck span is back to its previous duration.

Check every source rather than the one that recovered first. Full detail on post-deploy verification: Investigate deployments and CI failures.

Without CodeZero or Vörr

A coding agent can do much of this on its own if you set it up for it. Claude Code with gh, kubectl and a cloud CLI installed and credentialed, plus vendor MCP servers for Datadog and Sentry, can query metrics, read an issue's stack, tail pod logs, list workflow runs and read git history. ChatGPT or Claude chat can reason well over whatever you paste. The cost is in the setup and the stitching.

The specific gaps: each tool needs its own MCP server or CLI and credentials on the developer's machine; the service is checkout in your question, checkout-svc in metrics, web-checkout in the error tracker and deploy-checkout-prod in CI; building one timeline means querying every CI system separately; and an empty result looks the same whether the service is healthy, the name was wrong or the tool was never connected. Broad searches also cost context and steps, so an agent tends to stop at the first plausible explanation, usually the latest local commit.

With CodeZero and Vörr

One question, every source (Vörr)

Metrics, errors, logs, traces and deploys from more than 30 connected tools, with no observability credentials on the laptop. Stored credentials are never exposed to the assistant.

The right service (Vörr)

Names resolved through the service map and aliases, and every result labelled with its source, match quality and environment.

An honest zero (Vörr)

Coverage reports which sources can answer and why an answer might be wrong; a missing capability returns no connector.

The whole deploy timeline (Vörr)

Deploys across connected CI/CD sources in one feed, plus root-cause scoring that names the preceding deploy.

Code beyond the checkout (CodeZero)

Search, read and history across indexed repositories, including ones not cloned, with cross-repository edges and Verify.

A fix you approve (CodeZero)

A plan with repositories, phases and verification commands that the agent cannot approve itself, and confirmation before any edit outside the plan's repositories.

Used together, Vörr narrows the problem to a deploy, a service and a file from the running system, and CodeZero reads that file's history, checks its callers and carries the fix. Vörr is read-only towards your systems, so nothing in the investigation changes production.

Worked example: latency doubled, the wrong commit looks guilty

Checkout p99 doubled at 14:10. You open the checkout repository and the local log shows a commit from that morning touching the payment client. An agent reading only that history would explain how the commit could add latency and propose reverting it. Here is the same investigation with both products.

You: Using Vörr, is checkout degraded in production? Per source, last hour vs 7 days ago.

Vörr:
  Datadog APM  service=checkout-svc (explicit, prod)
    p99 840 ms (was 410 ms)   error rate 4.1% (was 0.6%)   throughput -3%
  Prometheus   job=checkout (explicit, prod)
    p99 812 ms (was 402 ms)   error rate 0.3% (was 0.2%)
  Sources disagree on error rate: APM counts each retried request as an error.
  Sentry       project=web-checkout (explicit, prod)
    top issue InventoryTimeoutError, 1,284 events, last seen 14:52
    culprit  checkout/clients/inventory.ts in reserveStock

You: Show the slowest checkout trace and the deploys before 14:10 anywhere in the org.

Vörr:
  Trace 7f3c...  total 1.9 s
    POST /checkout                      checkout-svc   1.9 s
      reserveStock -> GET /stock/:sku   inventory      1.6 s   <- bottleneck
        SELECT stock_levels ...         inventory-db   1.5 s
  Deploys 13:30-14:10 (prod)
    14:05  inventory   Argo CD          a91e2c4..d40b7f1
    11:52  checkout    GitHub Actions   payment client timeout refactor
  Root-cause scoring (checkout, last hour)
    dependency 86  database 71  deployment 64  network 18  cpu 9  memory 7
    preceding deploy: inventory 14:05

The checkout commit is real but its deploy at 11:52 is more than two hours before the change, and the bottleneck is in inventory. You take the inventory deploy into CodeZero.

You (CodeZero): In inventory, what changed in the stock lookup between a91e2c4 and d40b7f1?

CodeZero:
  inventory/src/stock/repository.ts  (last commit d40b7f1)
    getStockLevels now filters on LOWER(sku), which bypasses the
    (sku, warehouse_id) index on stock_levels.
  Callers of GET /stock/:sku (inferred from HTTP calls matched to the route):
    checkout   clients/inventory.ts:42   high
    fulfilment jobs/allocate.ts:88        medium
  Config: INVENTORY_DB_STATEMENT_TIMEOUT (set in deploy/helm/inventory/values-prod.yaml)

Worked example: conclusion, fix and verification

• Conclusion: the 14:05 inventory deploy made the stock query bypass its index. Checkout waits on it, times out and retries, which inflates the APM error count more than Prometheus's. Reverting the checkout commit would not have helped.
• Blast radius: fulfilment also calls the route, so the fix covers it. Raising the statement timeout in the Helm values would hide the symptom, not fix it.
• Plan: CodeZero plans a change in inventory only. A reproduction test asserts that the query uses the index, then the query is fixed to compare against a normalised column. Verification commands are the repository's test suite and the query-plan test. You approve the plan and the agent runs it; you review the working tree and open the PR yourself.
• Confirmation: after the fix deploys at 16:40, Vörr shows checkout p99 back to about 420 ms in both sources, the InventoryTimeoutError count flat since 16:43, and the inventory span back under 60 ms.

Follow-ups: verify the medium-confidence fulfilment edge in Graph, ask whether inventory's alert rules would have fired on the query latency, and add the alias deploy-checkout-prod to the service map if it was unmapped.

Verify and follow up

• Provenance: every Vörr figure should name its source, environment and match quality. Re-ask with an explicit service name when a match is name or unmapped.
• Coverage: an empty result is only a zero if Vörr's coverage says the source can answer. Ask why an answer could be wrong before you rely on a quiet signal.
• Agreement: when sources disagree, find the reason (retries, status-code mapping, sampling) before choosing which number to report.
• Code evidence: open the file CodeZero or Vörr points to and check the last-commit provenance against the deploy's commit range. Verify inferred cross-repository edges you are relying on.
• Plan verification: the reproduction should fail before the change and pass after. Run the declared verification commands yourself if the agent ran them unattended.
• After deploy: repeat the comparison window from Operation 11 in every source.

Next operations: if the cause was a missing alert or an unwatched service, set up checks with Monitor service health. If the fix needs coordinated changes in several repositories, follow Ship a cross-repository change. If this was an incident with people involved, carry the findings into From incident to fix.

Limits

• Vörr can only answer from tools you have connected. Signals from an unconnected tool are reported as missing, not inferred.
• Name resolution is only as good as the service map. Name, suggested and unmapped matches are labelled because they can be wrong, so confirm them before you rely on them.
• Root-cause scoring ranks candidates. It does not prove a cause; traces, deploys and code do.
• Vörr is read-only from the assistant. It cannot roll back, restart, silence an alert or change configuration. Those stay in your own tools.
• CodeZero indexes committed code on the default branch. Code that was deployed from another branch is only visible if it is checked out locally.
• CodeZero's references are matched by name and its callers are heuristic, not compiler-accurate. Cross-repository edges are inferred, which is why HTTP and config edges carry a confidence.
• Neither product reproduces production traffic. Reproduction depends on tests, data and environments you already have.

Related

• From incident to fix
• Investigate deployments and CI failures
• Monitor service health
• Where AI engineering gets stuck
• How Vörr AI and CodeZero work together
• Ask Vörr about your system
• Connect observability and CI tools
• Code Graph
• Chat
• Execution modes
• Coffee Mode
• Git Tree