Case study
How a review loop ships agent PRs
This is how my last team, at a US enterprise AI platform, got pull requests written by coding agents to production without a person babysitting each one. I was one of the engineers who built and ran the team's agent-first system. Below is the part I built myself, and what changed.
The problem
Agents could open a pull request in minutes. Getting it merged took far longer.
- Agent PRs sat waiting for a person to read the review comments and push fixes.
- CI was flaky. Infra failures turned checks red for reasons unrelated to the change, so people reran jobs by hand, and red checks got merged over.
- Every engineer prompted agents their own way, so the results were hard to predict and hard to review.
Writing code had stopped being the slow part. Everything after it had become the slow part.
What I built
Four pieces, each small enough to adopt on its own.
A review loop. When a review lands on a pull request, an agent reads the comments, fixes what it can, pushes, and asks for review again. It repeats until no serious issues are left, then hands the PR back to a person.
CI that reruns known infra failures. I rebuilt CI for agents: fast, predictable checks, and known infra failures rerun automatically instead of landing on someone's desk. A red check means something again, so there is no reason to merge over one.
Author-scoped auto-merge. A PR merges on its own only when it comes from an allowed author, touches none of the excluded paths, and carries a marker that its review is clear. Everything else waits for a person.
Shared skills. A library of agent instructions kept in the repos, so every agent works the same way on every engineer's machine.
What changed
- 7 engineers adopted the review loop, with about 1,700 runs.
- 5 repos got auto-merge, and 3 of them deploy to production on merge.
- 28 min to 93 s compile time in the data repos after rebuilding CI for agents.
- 45 to 1 a day bot PRs, after the flood was cut.
What a team should copy first
Don't start with auto-merge. Start where the waiting is.
- Make CI trustworthy. Fix or rerun known infra failures so a red check means a real problem. Nothing else holds up on flaky CI.
- Add the review loop. Let an agent handle the review comments on its own PRs, so a PR stops waiting for someone to push fixes.
- Write shared skills. Put agent instructions in the repo, so results stop depending on who ran the agent.
- Then turn on auto-merge, narrowly. One repo, a few authors, sensitive paths excluded, and a clear review required.