The question isn't whether an agent can write a function. It can, and it does, and most teams have a story about it writing a surprisingly good one. The question is whether an agent given a fully-specified ticket, the same tools a senior engineer has, and no supervision, lands a PR you'd actually merge.
That's a different problem. Writing code and shipping correct code diverge the moment the first test fails, the log says something unexpected, or the UI doesn't match the spec. A human debugs. An agent, without the right loop and the right tools, guesses and moves on.
This post is about what it takes to close that gap.
What "as good as a human" means right now
The most rigorous measure of autonomous coding capability is METR's time-horizon study. They measured the task length at which frontier models succeed 50% of the time, and found it doubling roughly every seven months. Claude 3.7 Sonnet sits at around 50 minutes per the arXiv paper; some coding tasks are already in the 14-hour range. That's real progress.
The SWE-bench Verified leaderboard tells a similar story — Claude Opus 4.5 scores 80.9%. But look at SWE-bench Pro, Scale's harder variant with tasks not yet public at training time: the same model drops to around 45.9%. The gap isn't a failure of the model. It's a failure of the evaluation methodology: Verified tests patch-generation on known issues; Pro tests actual problem-solving on unseen ones.
Both benchmarks measure a single shot at generating a correct patch. Autonomy is the loop, not the shot. An agent that fails once and has no mechanism to learn from that failure is not autonomous — it's a stochastic text generator that got lucky.
The autonomy pattern: Ralph loops
The Ralph loop is the simplest pattern that turns a stochastic generator into a system with a verifiable exit condition.
Geoff Huntley coined the term after observing that all effective autonomous coding setups share the same shape: a deterministic while-true loop, one task per iteration. The agent runs, an evaluator checks the output, and if it fails, the agent retries with the prior attempt's result in context. His follow-up post names it "Ralph Wiggum as a software engineer" — the character who tries, fails, and tries again, but crucially, tries again with what he learned.
Reference implementations exist: snarktank/ralph and vercel-labs/ralph-loop-agent both ship working versions of the pattern. The details differ; the shape is the same.
Why does this matter? Because it converts a probabilistic generator into a process that terminates on a verifiable condition. The agent doesn't have to get it right in one shot — it has to converge on a green evaluator. That's a tractable problem, not a gambling problem.
The honest critique is also worth stating: Ralph is brute force. Cost scales linearly with retry count. Convergence depends entirely on evaluator quality. If your evaluator is "did the tests pass?" and the tests only cover the happy path, the loop converges on code that passes the tests and ships the bug. The pattern is only as reliable as the signal it terminates on.
The toolchain that gives an agent eyes and hands
Every step of a human debugging session has an MCP equivalent.
A human opens DevTools, checks the network tab, tails the server logs, queries the database, sets a breakpoint, then re-runs the failing test. An agent with the right MCP tools does exactly the same sequence of operations — it just doesn't open a laptop.
Microsoft's Playwright MCP gives an agent the browser. Not screenshots — a structured accessibility tree, 20+ browser-control tools, the ability to click, type, navigate, and assert. Consult the Playwright MCP docs for setup. This is as close as any tool gets to "the agent actually opens the app and clicks around."
Logs are covered by MCP's logging spec. The MCP logging specification defines notifications/message with server-controlled severity levels. Combined with the tools in the modelcontextprotocol/servers reference repo, an agent can consume structured log output the same way a human reads tail -f.
Database access follows the same pattern. The reference servers repo ships Postgres and SQLite MCPs. The agent can query state at the moment of failure, compare against what the code expects, and find the divergence — the same move a senior engineer makes when the logs don't tell the whole story.
Codebase context is where the previous IBERANT post on repo AI readiness applies directly. An agent that doesn't understand your module boundaries, naming conventions, or test patterns will produce output that compiles and fails in production. AGENTS.md and CLAUDE.md are prerequisites, not optional decoration. Anthropic's own context engineering guide states this plainly: the agent's attention budget is finite, and signal-dense context outperforms comprehensive context every time.
The evaluator in the Ralph loop is the final piece. It can be npm test, a Playwright end-to-end suite, a typecheck, or all three in sequence. The only constraint: it has to be deterministic and machine-readable. "The build passed but there are warnings" is not an evaluator. "Exit 0 or exit 1" is.
A worked use case: autonomous bugfixing
Most real bug reports don't arrive as failing tests. They arrive as a sentence from a user: "the checkout button does nothing after I add a promo code." Maybe a screenshot. That's it.
The agent's first job is to turn that into a reproducible path — and with Playwright MCP, it can do exactly what the user did: open the app, navigate to the cart, enter the promo code, click the button, and observe what happens. The clickthrough path is the reproduction. No failing test required to start.
From there, the agent makes a judgment call. If the broken behavior is visible in the UI (wrong state, missing element, bad response rendered on screen), it writes a Playwright assertion that captures it. That becomes the evaluator for the Ralph loop — a script that fails until the bug is fixed. If the UI interaction points to a backend failure (a 500 in the network tab, a missing field in the API response, a log line that says something unexpected), the agent narrows the reproduction down to the API call that's actually failing. The Playwright script shrinks to a curl equivalent, or a direct call to the relevant endpoint. Simpler, faster, cheaper to run on every iteration.
Once the fix lands and the loop exits, the agent has all the material for a real test: the exact steps, the actual versus expected behavior, the layer where the failure occurred. It writes that as a proper automated test — not a throwaway script, but something that lives in the test suite and would catch a regression. Then it writes a short postmortem: what broke, where it broke, what the fix was, what the new test covers.
So the inputs needed are fewer than you might expect:
- A running dev environment the agent can reach. One command —
docker compose up,npm run devwith a seeded database. Not "ask the senior dev." - The user's description and, ideally, a screenshot or screen recording to anchor the clickthrough path.
- Read access to logs and network responses during the reproduction.
- A way to reset state between loop iterations so each attempt starts clean.
The loop then runs: reproduce via Playwright, inspect logs and API responses, identify the layer (frontend rendering, API contract, database state), narrow the reproduction to the smallest failing assertion, propose and apply a fix, re-run, evaluate. Green exits to a PR with the fix, the new regression test, and the postmortem. Red loops with the new evidence in context.
The hard requirements:
A stable environment is the prerequisite. If the app behaves differently on retry three because of leftover database state or a cached session, the loop can't converge. Isolation per iteration matters more than it seems.
A cost ceiling is non-negotiable. Set an iteration limit. The loop should abort, report where it got stuck, and hand back to a human — not spend indefinitely on a bug it can't close autonomously.
Human review sits at the PR boundary. The diff, the new test, and the postmortem all go to a human before anything merges. Autonomy ends there, not at main.
The failure modes are real: bugs that only reproduce on production data the agent can't access, concurrency issues the Playwright path can't trigger reliably, and bugs whose fix requires a judgment call outside the ticket scope. On that last one, the loop will converge on something — but it might be the wrong something if the evaluator accepts it. The Ralph loop doesn't protect against a misspecified task. It protects against a single shot at a well-specified one.
Where it gets complicated: the production data problem
The worked workflow above assumes one thing quietly: that the bug is reproducible with whatever data exists in the dev or QA environment. It usually isn't.
A user reports that submitting a multi-item order with a promo code applied to a subscription product fails silently. Your QA database has promo codes. It has subscription products. It does not have the specific account state, the billing edge case, or the combination of flags that triggers the failure in production. The Playwright path runs clean. The agent finds nothing. The loop exits green and the bug is still there.
To reproduce the bug faithfully, you need a copy of the database that mirrors what the user had when it failed. Which means a production snapshot. And that's where the operational complexity starts compounding.
The first problem is compliance. GDPR Article 32 requires appropriate technical measures to protect personal data — and copying real customer records into a dev environment that developers can query freely does not meet that bar. HIPAA, PCI DSS, and most national equivalents say the same thing with different specific rules. The fine for getting it wrong is not a slap on the wrist: up to 4% of global annual revenue under GDPR. Most companies that have a production snapshot in a dev environment are in violation and don't know it.
The second problem is size. A production database that reflects real usage is often hundreds of gigabytes or more. A developer laptop, a CI runner, or a per-branch dev environment cannot absorb that cheaply or quickly. The snapshot that takes four hours to restore is not useful as a per-iteration starting state for a Ralph loop. Isolation between iterations requires that each loop start from a clean, known state — and "restore 400GB before each attempt" is not a realistic approach.
The third problem is drift. A snapshot taken at incident time is correct once. The moment you restore it and the agent starts writing to it, the state diverges. Run the loop five times from the same snapshot and each run diverges a little more. Without a mechanism to reset to the exact production state before each iteration, reproducibility degrades as the loop runs.
These three problems — compliance, size, and drift — mean you can't just pg_dump production and call it done. The good news is there are real solutions, each making different trade-offs.
Database branching is the most ergonomic option if your infrastructure supports it. Neon implements copy-on-write branching at the storage layer: creating a branch from a production-like snapshot takes seconds regardless of database size, and each branch diverges independently. The agent's loop can reset to a clean branch before each iteration without a full restore. Xata takes a similar approach and adds built-in PII masking via pgstream, so the branch the agent sees has already had sensitive columns transformed before the first query. The compliance problem and the size problem both largely disappear.
Anonymized snapshots are the pragmatic middle ground for teams on traditional PostgreSQL or MySQL. The pattern: take a production snapshot, run a masking pass before any developer or agent can query it, then use the result as the reproducible base. Tonic.ai handles the masking step at the enterprise end — it preserves referential integrity and data distributions while replacing PII with realistic synthetic values, so the bug that triggered on a real edge case often still triggers on the masked equivalent. For open-source options, pgcopydb combined with postgresql-anonymizer gives you a fast copy and a declarative masking layer you can review in a pull request. The result is a snapshot that's safe to use in dev and representative enough to reproduce most data-dependent bugs.
Synthetic data seeding works when the bug is about patterns rather than specific records — an account that has been a customer for more than 36 months, or an order where the discount exceeds the shipping cost. Tools like MOSTLY AI and Gretel.ai generate statistically representative synthetic datasets that mirror the distribution of production without containing a single real customer record. You won't reproduce a bug tied to one specific user's account state, but you'll reproduce entire classes of bugs that never surface in handwritten seed data because nobody thought to write them.
Subset extraction is often overlooked. You rarely need the entire production database to reproduce a bug. You need the relevant records and their dependencies. pgcopydb supports filtered copies; a custom pg_dump with a --table and a WHERE clause isolating the affected account, order, or session is often enough. A 400GB database becomes a 200MB reproducible slice, which resets in seconds and can be committed to the repo as a seed fixture.
None of these remove the need for a deliberate decision about what data the agent is allowed to see. That decision has to be made before you build the loop, not after the first compliance incident. The loop's isolation boundary and the masking/branching strategy are part of the infrastructure, the same way the cost ceiling and the clean-state reset are. Build them in from the start.
IBERANT recommendations
These are the things worth doing before you run the first loop.
Wire up the full tool stack first, then write the loop. An agent without Playwright, logs, and database access is not autonomous — it's a patch generator with retry. The tools are not optional extras; they're the difference between the loop converging on a fix and the loop converging on a plausible-looking guess.
Treat your evaluator as the most important part of the system. A weak evaluator is worse than no loop, because it produces confident wrong answers. Before you run anything, ask: if the agent ships the wrong fix but the tests pass, would the evaluator catch it? If the answer is no, fix the evaluator first.
Start from real user reports, not synthetic tests. A bug described by an actual user, with a clickthrough path, is the right input. Don't reconstruct an artificial reproduction from memory. Let Playwright follow what the user did — that path is more honest than a test you wrote hoping to capture the failure.
Set a cost ceiling before the first run, not after. An iteration limit is not a pessimistic constraint; it's the thing that turns an infinite loop into a bounded process. Ten iterations is a reasonable starting point. If it hasn't converged by then, the problem is probably the evaluator or the environment, not the model.
Isolate state between every iteration. The agent will corrupt things — write partial data, leave open sessions, produce side effects that change the next run's outcome. A fresh database and clean app state per iteration is not gold-plating; it's what makes the results reproducible.
Don't skip the postmortem step. The regression test and postmortem the agent writes at loop exit are not ceremony. The test goes into the suite and catches the same class of bug next time. The postmortem gives the next engineer (or agent) enough context to understand why the fix is shaped the way it is. Both take seconds to generate and compound over time.
Keep your context files honest. AGENTS.md and CLAUDE.md have to describe how the codebase actually works, not how it worked six months ago. The loop won't compensate for a model working from stale context. Review these files after major refactors the same way you review tests.
Where the frontier actually is
For well-specified, locally-reproducible work with a real evaluator, autonomous agents already ship code that a human would have shipped. That sentence is not optimism — it's what METR's data and working Ralph-loop implementations show.
The bottleneck isn't the model. It's the infrastructure around it: reproducible environments, deterministic evaluators, scoped tool permissions, cost limits, and context-dense instruction files. The model has enough capability to close most bugs and implement most well-scoped features. The question is whether the toolchain around it gives it the same surface that a human has.
Most teams are not blocked on model capability. They're blocked on not having a seeded dev environment the agent can spin up, a test suite with meaningful coverage, or an evaluator that's worth trusting. Fix those things, and you're not waiting for a better model — you're running.
At IBERANT, these are the engineering problems we work on with the teams we build for. If you want to talk through what autonomous workflows would look like for your stack, reach out.