Shipping AI-Generated Code Safely - The CI/CD Guardrails That Make It Work
We have written before about how AI code review fits into a CI/CD pipeline - where to place it and what it misses. That piece ended on a point worth expanding into its own article: review is only as safe as the guardrails underneath it. When AI review lets a low-risk change ship without a human, the safety of the whole thing moves from the reviewer to the pipeline. This is that pipeline.
The shift is not subtle. Code volume has jumped as agents generate most changes on many teams, and the old assumption - that a human read and understood every line before it merged - no longer holds. If you remove the human from part of the flow, something has to take over the job the human was quietly doing: catching the change that is locally correct but globally wrong, and containing the blast when one slips through. That something is the pipeline, and it has to be built on purpose.
Why AI-generated code needs different guardrails
Human-written code came with implicit context. The person who wrote it knew why, remembered the last outage, and hesitated before touching the auth module. Machine-generated code carries none of that unless you make the context explicit. Three properties make it a distinct governance problem:
- Volume. More changes, more often, than any team produced before. Manual controls that scaled at ten PRs a day fall over at a hundred.
- No memory. An agent does not remember last quarter’s incident or the unwritten rule that nobody touches the billing service on a Friday. It optimises for the task in front of it.
- Plausible wrongness. AI output looks right. It compiles, it reads cleanly, the tests it wrote pass. The failure mode is not obvious garbage - it is confident, well-formatted code that quietly violates an assumption elsewhere in the system.
None of this means “do not ship AI code.” It means the controls that used to live in a reviewer’s head now have to live in the pipeline, where they run every time regardless of who - or what - opened the PR.
The five guardrails
Think of these as the layers that have to be in place before you let any change reach production without a human reading it. Each catches a different class of failure.
1. Tests as a hard gate
Unit and integration tests with real coverage, run on every pull request, blocking merge when red - no exceptions, no overrides, regardless of what the review bot said. This is the floor. When an agent writes both the code and the tests, add a check that the change actually adds meaningful assertions rather than tests that assert nothing; a green suite that tests the wrong thing is worse than no suite, because it signals safety that is not there.
2. Security and supply-chain scanning
SAST plus dependency scanning as a mandatory pipeline step. AI-generated code pulls in libraries as easily as a human does - and it will happily suggest a package that is outdated, abandoned, or (worse) a typosquat of the one you meant. Pin dependency versions, scan every change for known vulnerability classes, and fail the build on a critical finding rather than filing it for later.
3. Policy gates
Codified rules about what is allowed to merge automatically and what is not - enforced by the pipeline, not by convention. This is where “triage by blast radius” becomes real: the paths a change touches determine the gate it faces. A documentation edit and a change to the permissions model are not the same risk, and the pipeline should know the difference before a human is ever paged.
4. Provenance
A durable record of which change came from an agent, what prompt or task produced it, what context it had, and who or what approved it. When one agent’s output is another agent’s input, an untracked change becomes impossible to reason about after the fact. Provenance is what lets you answer “why is this code here?” three months later - and it is the single most-skipped guardrail, because nothing breaks the day you skip it.
5. Observability and a rollback that works
Post-deploy error metrics, alerting that fires in minutes, and an automated rollback - blue-green or canary with a trigger on error-rate threshold. This is the safety net for everything the first four layers missed. If a bad change does reach production, the question is not “will we catch it?” but “how fast, and can we undo it without a heroics-driven incident?” A rollback you have never tested is not a rollback.
Want to build guardrails for AI-generated code in your pipeline?
Book a free 30-min call
Blast-radius triage - the decision that ties it together
The five guardrails are the foundation; blast-radius triage is how you decide which changes lean on which guardrails. The core idea: classify every change by the damage it could do, and route it accordingly.
| Blast radius | Examples | Path to production |
|---|---|---|
| Low | Docs, additive tests, copy changes, isolated additive features behind a flag | Automated gates only (tests, scans, policy) - no human required |
| Medium | Internal API changes, non-critical service logic, dependency bumps | Automated gates + AI review; human optional based on policy |
| High | Auth, public API, permissions model, non-additive schema changes, payment paths | Automated gates + mandatory human review |
The classification itself should be automated - derived from the file paths a change touches, not from a developer self-selecting a label they can quietly downgrade. The pipeline reads the diff, applies the policy, and assigns the gate. That is the difference between a rule and a suggestion.
And the honest caveat: if your product is sensitive enough - payments, healthcare, safety-critical systems - the “low blast radius” category may be nearly empty, and that is a legitimate outcome. Triage does not mean forcing changes into an automated lane to hit a throughput number. It means being deliberate about where a human’s judgement is worth the wait.
Wiring it into the pipeline
Concretely, in a GitHub Actions flow the guardrails stack in order, with the policy gate deciding whether a human is required:
name: ai-code-guardrails
on:
pull_request:
types: [opened, synchronize, reopened, labeled]
jobs:
gates:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
# 1. Tests - hard gate, blocks on red
- name: Tests
run: ./scripts/test.sh
# 2. Security + supply-chain scan - fail on critical
- name: Security scan
run: ./scripts/scan.sh
# 3. Classify blast radius from changed paths
- name: Label blast radius
uses: actions/labeler@v5
with:
configuration-path: .github/blast-radius.yml
# 4. Provenance - record agent origin, task, approver
- name: Record provenance
run: ./scripts/provenance.sh
env:
PR_NUMBER: $
require-human:
needs: gates
if: contains(github.event.pull_request.labels.*.name, 'high-risk')
runs-on: ubuntu-latest
steps:
- name: Enforce human review for high-risk changes
run: |
echo "High blast radius - human review required."
exit 1
The require-human job fails deliberately for high-risk changes; paired with branch protection (a required status check), that blocks the merge until a person approves. Everything else clears the automated gates and can merge without a human - which is only safe because the gates above it are real. In Azure DevOps the same shape is built on branch policies: build validation runs the gates, and a path-scoped “minimum reviewers” policy enforces the human step for high blast-radius paths. Different mechanics, identical logic.
Note what the pipeline does not rely on: a developer remembering to add a label, or a reviewer choosing to look closely. The controls run every time, on every change, machine- or human-authored.
Governance at scale - where this connects to agent sprawl
One pipeline with good guardrails is a solved problem. The harder problem shows up when agents multiply across an organisation - different teams, different tools, each wiring its own automation with its own idea of what “safe” means. That is agent sprawl, and machine-generated code is one of its largest surfaces.
The guardrails in this article are the per-pipeline expression of a broader governance discipline: provenance so you can trace any change to its source; policy gates so the rules are enforced identically everywhere rather than reinvented per team; scanning so an agent in one corner of the org cannot quietly introduce a vulnerable dependency the rest of the organisation inherits. Without a shared standard, each team’s pipeline becomes a separate risk surface, and the blast radius stops being a per-change property and becomes an organisational one.
The practical move is to make these guardrails a template, not a per-team decision - a baseline pipeline configuration that every repository inherits, so “how AI code ships here” has one answer across the organisation rather than one per team.
When to invest in this - and when it is premature
Build these guardrails when:
- A meaningful share of your changes are agent- or assistant-generated, and volume is rising.
- You are already letting - or planning to let - some changes merge without a human reading them.
- You operate in a domain where a bad deploy has real cost: customer data, money, regulated workloads.
Hold off, or keep it lightweight, when:
- The team is small, PR volume is low, and a human still reads everything. Then the guardrails matter less than the review itself.
- You do not yet have the basics - tests, scans, a working rollback. Build those first. Layering policy gates and provenance on a pipeline with no test gate is decoration, not control.
The order matters. The failure we see most often is teams adopting autonomous code delivery for the speed, then bolting on guardrails after the first incident. It is far cheaper to build the pipeline first and let the agents run inside it.
How we can help
At Devopsity we build the pipelines that make autonomous code delivery safe rather than merely fast. That means the guardrails in this article wired in for real: tests and security scanning as hard gates, policy gates that enforce blast-radius triage from the diff, provenance for machine-generated changes, and observability with a rollback that has actually been tested. We also help standardise them across teams, so AI code ships the same safe way everywhere - not one improvised pipeline per squad.
If your teams are shipping more and more AI-generated code and you want the delivery process to keep up without turning production into a lottery - let’s talk about your CI/CD pipeline.
Shipping AI-generated code and want it safe?
Book a free 30-minute call. No pitch - a technical conversation about your delivery process.
Frequently asked questions
Is it safe to let AI-generated code merge without a human?
For low blast-radius changes, yes - but only when the pipeline underneath is solid: tests as a hard gate, security scanning, provenance, and a working rollback. The safety moves from the reviewer to the pipeline, so the pipeline has to earn it. For high blast-radius changes (auth, public API, schema, payments) a human stays in the loop regardless.
What is code provenance and why does it matter for AI code?
Provenance is a durable record of where a change came from - which agent or task produced it, what context it had, and who or what approved it. It matters because machine-generated code carries no implicit history. Without provenance you cannot answer “why is this here?” months later, and when one agent’s output feeds another, untracked changes become impossible to audit.
How do I classify the blast radius of a change automatically?
Derive it from the file paths the change touches, enforced in the pipeline - not from a label a developer picks by hand. Map sensitive paths (auth, migrations, public API, payment logic) to a high-risk gate that requires a human; treat additive and isolated changes as low-risk. Automating the classification is what stops it from being quietly downgraded.
Do these guardrails slow delivery down?
Done right, they speed it up. Low-risk changes clear automated gates in minutes instead of waiting in a review queue, while human attention concentrates on the few changes that can actually cause damage. The guardrails are what make it safe to remove the human from the low-risk path in the first place - without them, you either review everything (slow) or review nothing (dangerous).
Where do we start if we have no guardrails yet?
With the basics, in order: tests as a hard gate, then security scanning, then a rollback you have actually tested. Only once those hold should you add policy gates and provenance and start letting low-risk AI changes merge automatically. Guardrails layered on a pipeline with no test gate are decoration, not control.
