Putting AI in the Delivery Pipeline: What Actually Held Up in Production

Most “AI in your pipeline” writing is a demo. It works once, on a clean repo, in a screenshot. We run production platforms, so the only question we care about is different: after the novelty wears off and a tired engineer is shipping on a Friday, does the AI layer still earn its place, or does it quietly make things worse?

Over the last year we put five AI-driven pieces into live delivery pipelines: an AI pull-request reviewer, an agentic end-to-end test harness, agentic security testing in CI, LLM observability, and AI-assisted alert triage. Some of it paid for itself in a week. Some of it lied to us in ways that would have shipped bugs if we had trusted it. Here is the honest version.

1. The AI pull-request reviewer

What it does. Every pull request gets read by an LLM reviewer before a human opens it. It posts inline findings: logic it thinks is wrong, missing error handling, risky changes, style drift, obvious security smells.

How it helps. The value is not that it replaces a senior reviewer. It is that it clears the floor before the senior reviewer arrives. Trivial back-and-forth (unhandled null, a swallowed exception, a secret about to be logged) gets caught at minute zero instead of after a day in the review queue. On a busy repo that shortens the review cycle noticeably and gives your most expensive people a cleaner diff to reason about. It is consistent, it never gets bored on PR number forty, and it runs at 3am when nobody is awake to look.

Where it lied to us. Three things we had to engineer around, and you will hit all three:

• Confidence falls off a cliff with diff size. On a tight, focused PR the reviewer is sharp. On a 2,000-line change it produces confident, fluent, wrong commentary. The fix is social, not technical: smaller PRs, and treat a large-diff review as a hint, not a verdict.
• “Accepted as-is” is not approval. If you let the reviewer’s sign-off count as a passing gate, people learn to shrug past it. The AI opinion has to sit beside the human gate, never replace it.
• Findings can vanish between runs. A reviewer that reconciles its own thread will happily close a finding because “it did not reappear in the latest pass,” and a real security note can disappear with it. We do not let it auto-resolve anything that was ever tagged security. A finding is closed by a human or by a commit that fixes it, never by the model forgetting it said something.

Net: it is one of the highest-leverage pieces we run, precisely because we stopped asking it to be the decision and started using it as the first pass.

2. The agentic end-to-end test harness

What it does. An agent authors and repairs browser-level end-to-end tests (we use Playwright), and the suite gates the release train. A red run holds the promotion.

How it helps. The real win is reviving suites that have rotted. Every long-lived product accumulates a graveyard of E2E tests that were switched off because they were flaky. An agent is very good at the grind of getting them green again, updating selectors, and writing the boring happy-path and edge-case coverage nobody volunteers for. Once the suite actually gates the train, “is this release safe” stops being a feeling and becomes a check. That is the entire point of a pipeline.

Where it lied to us. The failure mode here is seductive because it looks like success:

• A test that cannot fail is not a test. The first agent-written passes were full of assertions that were green because they never exercised anything. A check against /thing/999 on an empty database passes forever and proves nothing. We now require every test to hit real seeded data and we measure that it can actually go red before we trust it green.
• Cascade settings bite. A naive propagate: true meant a flaky E2E run took down a downstream data-copy job with it. Gating is good; gating the wrong thing stops unrelated work.
• Flaky lists that rotate. If the set of “known flaky” tests completely churns every run, that is not flakiness, that is a real defect wearing a flaky costume. Reading the failure list over time mattered more than any single run.

3. Agentic security testing in CI

What it does. An agentic security tester runs against a live QA environment on a schedule and in the pipeline, probing for the classic web issues: broken access control, IDOR, auth bypass, injection.

How it helps. Continuous, cheap, and it never forgets to check. A full run costs a fraction of a coffee and finishes in under half an hour, so you can afford to run it often instead of once a year before an audit. For multi-tenant systems in particular it is worth its weight: authorization and cross-tenant leakage are exactly the bugs humans stop looking for after the hundredth endpoint, and exactly the bugs that end a company.

Where it lied to us. Two hard lessons:

• It only means something against real data and real auth. A security scan that cannot actually log in, or that probes IDs that do not exist, returns a clean report that proves nothing. The first pass has to be pointed at seeded accounts with real sessions, or the green is a lie.
• “Zero false positives” is a sales claim, not an engineering one. We adopt these tools and wrap them: triage the output, confirm the real ones, and keep proofs minimal. The tool finds, a human confirms. Anyone promising you a self-driving pentest is selling the demo.

4. LLM observability

What it does. Every call your AI features make to a model (the prompt, the response, latency, token cost, and whether it failed) is traced and searchable in one place.

How it helps. The moment you ship an LLM feature you have added a non-deterministic, pay-per-call dependency to production, and by default it is a black box. Observability turns it back into something you can operate. When a feature “feels dumb this week,” you can read the actual prompt that went out. When the bill jumps, you can see which feature and which prompt drove it. When calls fail, you see the pattern instead of guessing. It is the difference between running an AI feature and merely hoping it still works.

Where it is just plumbing. This one did not lie to us; it is the least glamorous and the one we would install first. You cannot improve, cost-control, or debug what you cannot see.

5. AI-assisted alert triage

What it does. Instead of a wall of raw alerts, incoming signals get grouped, deduplicated, and summarised, with a suggested likely cause and the context of similar recent incidents.

How it helps. Alert fatigue is a reliability risk in itself: when every pod restart pages someone, people stop reading pages, and the one that matters gets missed in the noise. Collapsing a storm of forty correlated alerts into one summarised incident with a probable cause is the difference between a calm response and a 2am scramble. It does not fix the outage; it makes the human faster at fixing it.

Where we drew the line. The triage layer recommends, it never decides, and it is never allowed to silence an alert on its own. An AI that can auto-resolve its own pages will eventually auto-resolve the real one. Suggesting is safe. Suppressing is not.

The rule underneath all of it

Every one of these pieces failed in the same shape at least once: it produced a confident green that proved nothing. The AI reviewer closing its own findings, the E2E test that could not fail, the security scan with no real session, the triage that quietly swallowed a page. The through-line is a single rule we now apply everywhere:

A check that cannot fail is not a check, and a machine’s confidence is not evidence.

So we keep the human on the gate that matters, we point every automated check at real data and prove it can go red, and we treat AI output as a strong first pass rather than a verdict. Inside those rails, the delivery pipeline genuinely got faster, safer, and cheaper to run. Outside them, AI tooling does the most dangerous thing a tool can do: it makes a broken pipeline look like it is working.

If you are adding AI to your own pipeline, start with observability so you can see what is happening, add the PR reviewer and E2E harness for leverage, wire in security testing against real data, and keep a human holding every gate that can hurt you. That ordering is not an accident. It is what held up.