Investigate deployments and CI failures

When you'd do this

• A build or pipeline is red and the log is too long to read, or the failing step is several stages away from the actual cause.
• A job fails on main but passes on your branch, or fails one run and passes the next, and you need to know whether it is the code, the environment or a flaky test.
• An Argo CD or Flux CD application is degraded or stuck after a sync, and you need to know which change and which configuration did it.
• Someone asks what deployed in the last 24 hours, across every team and tool, before an incident review or a release freeze.
• A deploy went out and you need evidence that it is healthy, or evidence that it is not, before deciding whether to roll back or fix forward.
• Staging works and production does not, and you suspect the environments have drifted apart in branch, variables or deploy config.

If the deploy succeeded and the problem is runtime behaviour, such as errors or latency, the full investigation is in Debug a production error. If an incident is open and people need coordinating, start with From incident to fix.

Before you start

• Vörr: your organisation set up and repositories connected. GitHub Actions, GitLab CI and Bitbucket Pipelines come through the code connection; Jenkins, CircleCI, Buildkite, Drone, Argo CD, Flux CD, Devtron and other CI/CD tools are connected separately. For Jenkins, use an API token rather than SSO web credentials.
• Vörr: important branches mapped to environments (for example main to production, develop to staging), so a deploy and a commit can be placed in the right environment.
• Vörr: jobs, pipelines and applications mapped to service names in the service map by an administrator, so deploy-payments-prod, payments-api and the payments Argo CD application resolve to one service. Unmapped names still work, but matches are labelled name or unmapped.
• Vörr: observability tools connected if you want post-deploy verification, and the connector added to Claude, ChatGPT or another MCP assistant with your own consent. ChatGPT needs a paid plan with Developer mode turned on.
• CodeZero: installed in VS Code with Claude Code, Codex, GitHub Copilot or Cursor as the agent, and your organisation's repositories indexed, including the ones that hold deploy config such as Helm charts and Kubernetes manifests.

If any of this isn't set up yet, start here:

• Connect observability and CI tools
• Connect repositories and map branches
• Install and connect CodeZero

How the work splits

Vörr reads the pipeline and the running system from your assistant: build-failure diagnosis, the organisation-wide deploy feed, deploy and health status, regression windows it detects (with Grafana connected) shown alongside deploys, and post-deploy signals, each with its source. CodeZero works on the code: it searches across repositories, compares the last good commit with the failing one, finds the configuration a step reads, and carries the fix through a plan with verification commands. They are separate products, and you move findings from one to the other. Neither one triggers, retries or rolls back a pipeline. Those actions stay in your CI/CD tools and with the people who own them.

Operation 1: Diagnose a failed build

1. Ask Vörr about the failing job by the name you use. It resolves the name through the service map and aliases, and can list the real job and pipeline names in each connected CI tool if the name is ambiguous.
2. Vörr reads the build logs of the most recent few failed builds from the connected CI source and returns a build-failure diagnosis for each: the failure category, the phase it failed in (for example checkout, dependency install, compile, test, package or deploy), the reason, the relevant log excerpt, and, from Jenkins, CircleCI, Buildkite or Drone where the build produced a test report, the names of the failing tests.
3. Check that the answer names the CI source, the job, the build number and the excerpt it relied on. Ask it to say what it cannot conclude from the log.

A good diagnosis points at the first real error, not the last line of the log. A test step that fails because the build step produced nothing is a build failure, and the excerpt should show that. If the CI tool is not connected, Vörr says so instead of returning nothing.

Operation 2: Decide whether it is the code, the environment or a flaky test

1. Start from Vörr's category and phase. Compile errors, type errors and assertion failures in a changed area point at code. Registry timeouts, missing credentials, out-of-memory kills, full disks and unreachable services point at the environment or infrastructure.
2. Ask Vörr whether recent builds of the same job failed the same way, where the CI source returns run history, and whether other pipelines failed in the same window. A wave of failures across unrelated repositories at the same time is rarely a code change.
3. For a suspected flaky test, compare the failing tests across runs: a test that fails on unrelated commits and passes on retry, with a timing or ordering error, is a flake candidate. Treat it as a lead and confirm it by reading the test in CodeZero.
4. If the failure is environmental, check the infrastructure the runner depends on with Vörr, for example restarts or memory pressure on self-hosted runners in Kubernetes.

Keep the three outcomes distinct in what you report. Retrying a code failure wastes a run, and patching code for an environment failure hides the real problem. If the evidence does not separate them, say so rather than picking one.

Operation 3: Find the change behind the failure

1. Take the failing build from Vörr, and the last good build and commits where the CI source provides them; otherwise read them from your CI tool or Git Tree.
2. In CodeZero's Git Tree, compare the two commits or branches to see exactly what changed between them. Use commit details to read each change.
3. Use CodeZero's recent changes for the repository or for the path the failing step points to, then Blame and file history on the specific lines.
4. Ask CodeZero for the owners of the path by commit history, so you know who to ask before you change it.

The change behind a build failure is not always in the application repository. A shared CI template, a base image or a dependency bump in another repository can break a build without any commit in the failing one. If the diff is empty or unrelated, look outside the repository.

Operation 4: Find the code and config the failing step points to

1. From the excerpt, take the file, function, test or error message and search for it in CodeZero across all indexed repositories, not just the one you have open.
2. Use CodeZero to find the settings the code expects and where each one is set, including .env.example, to see which ones the failing step needs.
3. Find where those settings and the deploy config are actually set. Helm values, Kubernetes manifests and Compose files often live in a separate deploy or platform repository.
4. Ask Vörr for a config-to-code investigation when the failure is a missing or invalid setting. It uses the same code index to connect the setting to the code that reads it.

Open the file you are about to change and check its last-commit provenance. CodeZero's references are matched by name, so a same-named function in another service can appear in the results.

Operation 5: Fix through a plan with verification

1. In CodeZero, describe the cause with the evidence: the job, the build, the excerpt, the commit range and the file. Use Planned mode if you want no code changes until you approve a plan.
2. Ask the agent to reproduce the failure locally first, for example by running the failing test or the build step, and to put that in the plan.
3. The plan names the repositories it touches, its phases and the verification commands. If the fix belongs in a deploy repository you have not cloned, CodeZero offers to clone it. Edits in the plan's repositories go ahead, an edit in another checkout of an organisation repository asks first, paths outside every checkout are refused, and the agent cannot approve its own plan.
4. After approval, only the declared verification commands run without asking. CodeZero commits each finished task locally; you review the commits and push yourself, and your CI runs the pipeline again.

A good plan has a reproduction that fails before the change and passes after it, and verification you can run yourself. If the agent asks a clarification question, such as whether PAYMENTS_REGION should be required in every environment, answer it rather than letting it assume.

Operation 6: Investigate a failed or degraded rollout

1. Ask Vörr for the deploy and health status of the application from the connected CD tool, for example Argo CD, Flux CD or Devtron: whether the sync or release succeeded, which revision it is on and whether it reports healthy.
2. For a Kubernetes workload, ask Vörr for infrastructure health: restarts, pods ready, CPU and memory. A rollout that never becomes ready often shows crash loops or out-of-memory restarts here.
3. Ask for release health where the error tracker provides it, to see whether the new release is producing new errors.
4. Take the revision and the failing resource into CodeZero to find the manifest, chart or values change that went with it.

Check the environment on every answer. An application that is degraded in staging after the same revision is healthy in production points at environment configuration, not at the code.

Operation 7: See what deployed across the organisation

1. Ask Vörr for its deploy feed for a time window. It brings deploys from every connected CI/CD source into one timeline: service, environment, source, time, and the commit or revision where the source reports one.
2. Filter by environment or service, or ask for everything before an incident review.
3. With Grafana connected, ask for the regression windows Vörr detects, shown alongside the deploys in the same period.

This is the question a local checkout cannot answer: the deploy that matters is often in another service. Debug a production error has a worked example.

Operation 8: Verify a deploy after it ships

1. Confirm in Vörr's deploy feed what went out, where and when, with the commit or revision where the source reports one.
2. Ask Vörr for error rate, p95 latency (plus p50 and p99 where the source reports them) and throughput for a window after the deploy, compared with the window before, per source. Vörr does not average sources together, and flags them when they disagree.
3. Ask for any regression window Vörr detects (with Grafana connected) around the deploy, for the top new errors since the deploy, and for the service's current SLO status, with remaining budget where the source reports it (for example Dynatrace).
4. Repeat the check after enough traffic has passed to make the window meaningful.

A quiet signal only counts if the source can answer. Ask Vörr's coverage which sources watch this service and whether a silence or a failing alert rule could hide a problem.

Operation 9: Check whether the deploy rolls back cleanly

1. Before anyone rolls back, check in CodeZero whether the change includes anything that does not roll back cleanly, such as a schema migration or a message or API contract change. The data layer in CodeZero's Graph shows which code reads and writes the tables involved.
2. Make the rollback or fix-forward call yourself, or with the service owner, and carry it out in your CD tool or with git. Neither Vörr nor CodeZero rolls back a deploy.

The full rollback-or-fix-forward comparison is in From incident to fix.

Operation 10: Find deploy config drift between environments

1. Ask Vörr which branches map to which environments for the repository, and which commit or revision each environment is running according to the deploy feed, where the source reports one.
2. In CodeZero, compare the deploy config for each environment: Helm values, manifests, Compose files and CI workflow variables, across the repositories that hold them.
3. Compare the settings the code expects with what each environment sets. A variable that is set in staging and missing in production is the classic cause of a works-in-staging failure.

Drift found in config files is only what the repositories declare. Values set by hand in a CD tool or a secrets manager do not appear in the code index, so confirm the live value in the tool itself.

Without CodeZero or Vörr

A coding agent can do a fair amount of this alone. Claude Code with gh installed and authenticated can list GitHub Actions runs, read failed job logs, re-run failed jobs and read the diff between two commits. With the Argo CD CLI, kubectl, a Jenkins or CircleCI CLI or MCP server, each credentialed on the laptop, it can read those tools too. ChatGPT or Claude chat can reason well over a pasted log. The cost is in setup, stitching and coverage.

The specific gaps: several CI/CD systems each need their own access; the job is called deploy-payments-prod in Jenkins, payments in Argo CD and payments-api in the repository, and nothing maps one to the other; an organisation-wide deploy timeline means querying every system; and an empty result looks the same whether nothing deployed, the name was wrong or the tool was never connected. One thing the agent alone can do that Vörr cannot: re-run a GitHub Actions job with gh. Vörr is read-only.

With CodeZero and Vörr

The failure, already read (Vörr)

A build-failure diagnosis of the most recent few failed builds, with category, phase and reason from Jenkins, CircleCI, Buildkite, GitHub Actions, GitLab CI, Bitbucket Pipelines and other connected CI tools, failing test names from Jenkins, CircleCI, Buildkite and Drone where a test report exists, and no CI credentials on the laptop.

One deploy timeline (Vörr)

Deploys from every connected CI/CD source in one feed, with environment, source, and the commit or revision where the source reports one, with the regression windows Vörr detects (with Grafana connected) shown alongside.

The right job and service (Vörr)

Job, pipeline and application names resolved through the service map, with source, match quality and environment on every result, and no connector instead of a silent zero.

The change and its config (CodeZero)

Compare last good and failing in Git Tree, recent changes and owners, and search, settings and deploy config across indexed repositories, including ones not cloned.

A fix you approve (CodeZero)

A plan with repositories, phases and verification commands that the agent cannot approve itself, and confirmation before any edit outside the plan's repositories.

Evidence after it ships (Vörr)

Comparison windows per source, current SLO status with remaining budget where the source reports it, and release health for the deploy you just made.

Used together, Vörr tells you what failed, what shipped and how it is behaving, and CodeZero tells you which change and which configuration did it and carries the fix. Nothing in the investigation changes a pipeline or a cluster.

Worked example: a red build that wasn't the code

The payments build on main has failed twice since this morning. The latest commit touches the config loader, so the obvious guess is that the commit broke it.

You: Using Vörr, why did the latest payments build fail? Show what you relied on.

Vörr:
  Source   Jenkins  job=payments-ci (explicit, maps to payments)
  Build    #2187 on main @ 8f27e44   failed 09:42
  Category test failure        Phase  test
  Reason   ConfigLoader rejects an empty region at startup
  Excerpt
    [test] ConfigLoaderTest > loads_defaults FAILED
    [test]   ConfigError: region must not be empty (PAYMENTS_REGION)
    [test]   at ConfigLoader.validate (src/config/loader.ts:57)
  Failing tests  ConfigLoaderTest.loads_defaults, PaymentClientTest.init
  Previous build #2186 @ 3c1d9a0 failed in the same phase with the same error
  Last success   #2185 @ 3c1d9a0 at 17:10 yesterday
  Cannot conclude: whether PAYMENTS_REGION was ever set in the CI environment.

The same commit, 3c1d9a0, passed yesterday and failed this morning, so the latest commit is not the cause. Something outside the repository changed between the two runs. You take that into CodeZero.

You (CodeZero): ConfigLoader.validate throws 'region must not be empty'. Where does the test
environment get PAYMENTS_REGION, and what changed there since yesterday 17:00?

CodeZero:
  payments/src/config/loader.ts:57  validate() requires PAYMENTS_REGION
    default: none (since 3c1d9a0, 2 days ago)
  payments/.env.example  PAYMENTS_REGION=eu-west-1
  Set for CI in: ci-templates/jenkins/node-service.groovy  (shared template)
    commit b71e0c2 at 08:55 today: 'move region vars to credentials store'
    removed the PAYMENTS_REGION default from the test stage
  Other jobs using this template: orders-ci, ledger-ci
  Owners of ci-templates/jenkins by commit history: platform team

Worked example: conclusion, fix and verification

• Conclusion: an environment failure, not a code failure and not a flake. A change to the shared Jenkins template removed a variable the payments tests rely on. A retry would have failed again, and reverting the config loader commit would have hidden a real requirement.
• Check: you ask Vörr whether orders-ci and ledger-ci have failed since 08:55. Orders has, with the same reason, which confirms the template as the cause.
• Plan: after asking the platform team, CodeZero plans one change in ci-templates to restore a test-stage default, plus a test fixture in payments so the unit tests do not depend on CI variables. Verification commands are the payments unit tests and the template's lint check. You approve the plan, review the commits and open the pull requests yourself.
• After merge: Vörr shows payments-ci build #2190 and orders-ci passing. The deploy feed shows payments reaching staging at 11:05, and the post-deploy comparison shows no change in error rate or latency.

Follow-ups: ask the platform team to add the region variables to the credentials store for the test stage, and map ledger-ci to the ledger service in the service map, since Vörr had matched it by inference.

Verify and follow up

• Provenance: every Vörr answer should name the CI or CD source, job or application, build or revision, and environment. Re-ask with an explicit name when the match is name or unmapped.
• Excerpt: read the excerpt Vörr relied on and check that it is the first real error. Open the full log in your CI tool if the excerpt and the reason do not obviously agree.
• Coverage: an empty deploy feed or a passing check only counts if Vörr's coverage says the source can answer for that service.
• Code evidence: check the last-commit provenance of the file you change against the commit range of the failing build, and confirm name-matched references by reading the call site.
• Plan verification: the reproduction should fail before the change and pass after. Then let your CI run the pipeline; neither product runs it for you.
• After deploy: repeat the post-deploy comparison per source once enough traffic has passed.

Next operations: if a deploy passed but the service misbehaves, continue in Debug a production error. If the failure showed a service nobody watches, set up checks in Monitor service health. For a pull request that should have caught this, see Review a pull request.

Limits

• Vörr cannot trigger, retry, cancel or roll back a pipeline or deploy, and it cannot change CI variables or CD configuration. It is read-only from the assistant.
• Vörr only sees CI/CD tools you have connected. A deploy from an unconnected tool or a manual kubectl apply does not appear in the deploy feed.
• Build-failure diagnosis is only as good as the log. If the log is truncated, or the real error is in an artifact the log does not include, Vörr says what it could not conclude.
• Telling a flaky test from a real failure is a judgement based on run history. Vörr and the agent can show the pattern; they cannot prove a test is flaky.
• Name resolution between jobs, applications and services depends on the service map. Name, suggested and unmapped matches are labelled because they can be wrong.
• CodeZero indexes committed code on the default branch. A failing feature branch is only visible if it is checked out locally, and values set by hand in a CD tool or secrets manager are not in the index.
• CodeZero's references are matched by name and its callers are heuristic, not compiler-accurate.

Related

• Debug a production error
• From incident to fix
• Monitor service health
• Review a pull request
• Where AI engineering gets stuck
• Connect observability and CI tools
• Connect repositories
• Ask Vörr about your system
• Git Tree
• Multi-repository work
• Execution modes
• Coffee Mode