Four of Five Bugs in AI-Built System Escaped Automated Testing, Study Finds

TL;DR

An Amazon researcher had the AI model Claude build a small multi-part software system in one long session, then catalogued every bug it introduced. Four of the five bugs it made were never caught by a test — they only surfaced when something broke while running. Separately, researchers at Concordia University found that AI-written code has different problems at different scales: small isolated functions get bloated, overly long methods, while full multi-file projects get worse structural problems, and writing more detailed instructions didn’t fix that. Neither study claims these findings generalize beyond the specific agents and setups tested.

What happened

An applied scientist at Amazon, Phanindra Reddy Madduru, gave a single instance of Claude — run through a command-line coding tool — one extended session to build a fixed system: a persistent graph database, a vector search index filtered by metadata, a multi-stage pipeline that ingested documents asynchronously, and a client that rendered the result as an interactive graph diagram, against a specification that was already written, with the agent free to make only narrow interaction-design choices.

Over that session the agent introduced, and later fixed, five defects spanning four different kinds of system-level failure:

• A reliability bug: the app located its config files relative to whatever directory the process happened to launch from, so it broke outside the default setup.
• A storage bug: the save function generated a new random ID every time, violating a requirement that repeating the same save should not create duplicates.
• A consistency bug: a graph-traversal function fetched connected nodes in only one direction while claiming both, crashing an unrelated visualization client and recurring at a second place in the code.
• An efficiency bug: the ingestion pipeline made its network calls one after another instead of in parallel, so processing time grew with document size.
• A rendering bug: a formula sizing on-screen labels had a floor value that broke the math meant to keep labels a constant size at any zoom level.

Of these five, only the storage bug was caught by an automated test, one added after the fact; the other four surfaced only through runtime symptoms — a crash, a runtime exception, a user noticing slow performance, or an automated screenshot comparison — with no static or type-level warning at all.

The efficiency bug illustrates the gap. It was flagged after an 18-chunk document took 242 seconds to process. The agent parallelized the slow pipeline stage, tested the fix only on a smaller document, and reported it as working without ever re-running the original 18-chunk document to confirm the number had actually improved. Madduru writes that this was “a claimed performance fix was never re-measured on the regression that motivated it — an unverified engineering claim that the self-reported development loop, in this instance, had no mechanism to catch.”

The same paper also benchmarked the system’s retrieval design: on 100 questions from the HotpotQA dataset checked against a pool of 2,994 paragraphs, restricting candidates to a graph-identified set of entities found the correct evidence 100% of the time from the third candidate onward, while ordinary semantic search across the whole pool found it only 69% of the time even after ten candidates — a gap that held at every candidate count tested. The author states plainly that none of this establishes how often these defects would recur with a different agent, a smaller model, or a second session. The paper’s byline lists Madduru as affiliated with Amazon.com.

A second paper looked at the same question with a different method. Researchers at Concordia University prompted five models — Gemini 2.5 Pro, Llama 3.3 70B, DeepSeek-Coder-V2 16B, and Qwen3-Coder at 30B and 480B parameters — on 90 algorithmic problems, then used the multi-agent framework MetaGPT, built on Qwen3-Coder 480B, to generate five complete multi-file software projects, scoring everything with the static-analysis tool PyExamine.

On the isolated problems, the largest model, Qwen3-Coder 480B, produced 11 instances of a “Long Method” smell — a single function grown too large to review easily — versus 1 in the human baseline, while the human code alone showed a different pattern, “Temporal Field” smells, absent from every model’s output. Giving models coding-standard examples beforehand did not reduce the Long Method problem and in some cases worsened it: Qwen3-Coder 480B’s count rose from 11 to 13.

Once the same models built complete systems through MetaGPT, the dominant defect type changed: method-level bloat gave way to structural problems — logic concentrated into oversized manager classes, code copy-pasted inline instead of reused through helper functions, and code split across files without real separation of responsibility between them. The authors write that “file separation does not equate to logical separation.” Across those projects, total lines of code correlated almost perfectly with how many architectural smells appeared, while how detailed the prompt’s requirements were had no measurable effect on any smell category. The authors summarize: “AI does not eliminate flaws but rather introduces a distinct “machine signature” of defects.” They add that “current agents operate as proficient “junior developers”: they follow instructions with syntactic precision but lack the “senior architect” foresight needed to manage system-wide dependencies and maintain a coherent architectural vision.”

The authors caution that the MetaGPT findings are specific to that framework’s workflow design rather than evidence about multi-agent coding systems broadly, though they consider the pattern likely indicative of wider challenges. They also note the algorithmic-problem experiment tested only each model’s first attempt, with no chance to revise using test-case feedback.

What this means (and what it does not)

Together, the two papers describe the same shape of problem measured two different ways: in the Amazon case, most of the bugs an agent introduced only showed up once the system was running, not while it was being written or reviewed by a test suite. In the Concordia study, the defect type shifted from small, fixable method bloat to structural problems that more detailed instructions did not fix.

Amazon employs the case study’s author and sells its own LLM coding-agent products; a candid accounting of an agent’s limitations can also read as evidence of self-scrutiny around that category, even though the paper itself is critical of the agent’s performance. Anthropic, which makes Claude, and the vendors evaluated in the Concordia study — Google, Meta, Alibaba, DeepSeek — are not authors of either paper but have a general commercial stake in how their models’ reliability at building systems is perceived. The maintainers of MetaGPT likewise have a stake in whether readers treat the architectural-decay finding as specific to their framework or as a claim about agentic coding tools generally; the study’s own authors caution against the broader reading.

What does not follow from either paper is that these exact defect rates, or this exact taxonomy, apply to other agents, other models, or other coding frameworks. Both papers are explicit that they tested one setup each — one Claude session in one case, five specific models and one multi-agent framework in the other — and neither claims its numbers travel beyond that.

What we still do not know

Whether the Amazon case study’s five-defect pattern recurs in other sessions, with other agents, or with smaller and less capable models than the one used — the author explicitly declines to claim it does.

Whether the code smells the Concordia study measures with static analysis are the same kind of problem as the runtime defects the Amazon study catalogs — idempotency violations, crashes, latency regressions. The two papers measure different things, and neither claims otherwise.

Whether agent frameworks other than MetaGPT — the Concordia authors name Cursor Composer and GitHub’s SpecKit as untested alternatives — show the same architectural-decay pattern.

Whether giving agents an iterative debug loop with test-case feedback, deliberately excluded from the Concordia study’s first experiment, would reduce the architectural defect rates it measured.

Whether the Amazon study’s 69% recall figure for unfiltered retrieval would hold if an agent actually performed the entity identification itself, rather than the gold-label substitute used in its place — the author flags this as the main threat to generalizing that number.

Neither paper has been peer-reviewed. Both are unreviewed preprints; one was posted a day before this reporting was compiled.