Do AI-written tests catch AI-written bugs? A common-cause experiment with mutation testing

In our article on verification debt we put forward a hypothesis that, if true, would have uncomfortable consequences. We called it H3, common cause: tests written by an AI fail to catch the bugs in code written by that same AI, because both share the same blind spots. If that happens, generating more tests with the same AI does not lower the risk. There is a floor that does not break.

We also said it was the cheapest result to produce. So we produced it: a small experiment, with the design and the success criterion written down before a single line of code was generated.

The result, up front: in this setting, the experiment did not support the hypothesis. All four test suites caught between 97.9% and 99.7% of the seeded bugs, and the comparison we fixed in advance did not reach the threshold. A small pattern did show up, though, and it is worth telling: the two suites written by the same model that wrote the code let exactly the same bugs through in one module, whether or not they had seen the code.

The question

If whoever writes the code and whoever writes the tests share a bias, a bug can slip past both filters at once. Knight and Leveson measured this in 1986 with programs written independently by people: they failed in the same places. With AI the suspicion is stronger, because the same model can write the code and its tests in the same conversation.

There are two possible sources of that shared bias, and it helps to separate them:

• Sharing the model: the same model makes the same mistakes, or the same omissions, when coding and when testing.
• Seeing the code: a tester who looks at the implementation tends to confirm what the code does rather than what it should do. If the code has a bug, the test "freezes" it as expected behavior.

The design

Twelve small Python modules, each with a written specification and explicit edge cases: Luhn validation, semantic version comparison, interval merging, banker's rounding, line wrapping, Roman numerals, a token bucket, quoted CSV fields, ISO 8601 durations, date range overlap, a password policy and Mexican phone number normalization.

A subagent running the Opus model, with a clean context, wrote all twelve implementations from the specifications alone. Then four different subagents each wrote one test suite, with identical instructions except for what they were allowed to read:

As an oracle, we hand-wrote a reference implementation of each module before generating anything, and checked it against the 69 examples in the specifications. No subagent could see it.

Metrics, in order of importance:

1. Real defects caught. We compared the generated code against the reference with thousands of random inputs. Each discrepancy is a real defect, and we measure what share each suite catches.
2. Mutation score. We seeded small bugs into the generated code (a < turned into <=, an and into an or, one constant into another) and measured how many each suite detects.
3. Tests that assert something false: those that fail against the reference.

A suite's mutation score is the share of distinguishable mutants it detects:

To compare two suites module by module we use the sign test: if one detects fewer in g of the n untied modules, the probability of a result at least as extreme by pure chance is

Criterion fixed in advance. The hypothesis would be supported if suite A caught fewer bugs than suite D in at least 10 of the 12 modules, or by 15 points or more overall. It would be refuted if the difference went the same way in fewer than 9 modules. And if the generated code turned out to have no real defects, the first metric would have nothing to measure: we would say so and use the second.

Result 1: the code had no defects

The twelve generated implementations had no discrepancy with the reference across 20,000 random inputs per module: 240,000 cases in all.

Before believing it, we checked that the comparison tool was not blind: we planted bugs on purpose in a copy of the reference, and it caught them in every module tested, with hundreds of discrepancies each.

So the main metric was left with nothing to measure, as we had anticipated might happen. That is a finding in itself: with precise specifications and small modules, the author model left no bugs that random inputs could find. It is consistent with rival explanation R3 from the first article: AI code can have high baseline quality, at least on well-scoped tasks.

Result 2: all four suites catch almost everything

Of 408 mutants, 383 can be told apart from the reference. The rest are equivalent or beyond the reach of the comparison tool, and are excluded.

• A versus D, the preregistered comparison: A detects fewer in 2 modules, more in none, and ties in 10. The overall difference is 1 point. The sign test gives p = 0.25. By the criterion we fixed, the experiment does not support the hypothesis.
• Seeing the code made no measurable difference. A and B (same model) are less than a point apart, and C and D (other model) tie.
• No test asserted anything false. All 765 tests across the four suites pass against the reference. No bug got "frozen", though to be fair there were no bugs to freeze.

Result 3, exploratory: the same model's blind spot

This analysis was not preregistered, so we present it as exploratory.

Suites A and B, both written by the same model that wrote the code, let exactly the same three mutants through in the interval-merging module, one having seen the code and the other not. All three do the same thing: when merging, they compare against the first interval in the result instead of the last. The bug only shows up when there are at least two separate intervals before one that merges with the last, and neither suite tested that case. The other model's suites did, and caught all three.

That is the form of common cause the hypothesis describes, on a small scale: the same model, with or without the code in front of it, left out the same case. In total, the same-model suites let 13 mutants through and the other model's suites let 2 through.

Before giving it weight, there is a rival explanation we cannot rule out: the other model wrote more tests (410 versus 355). It may be thoroughness rather than independence.

Two more findings:

• A round trip can prove nothing. Suite B, the only one with no hand-written expected values for Roman numerals, let four changes to the value table through (for example, D worth 501). It relied on the property the specification suggested: converting to Roman and back must return the same number. But the generated code builds one conversion from the other, so if the table is wrong, both directions are wrong in the same way and the property still passes.
• One case nobody tested. A single mutant survived all four suites: accepting a line width of 0 when the text is empty.

A bug in our own verification

The first version of this analysis had a bug, and we found it by checking a result that did not add up. The runner executing the suites did not record failures inside subTest, which several suites use to group cases: a test with a failed subcase vanished from the results instead of counting as failed. With that bug, scores came out between 90.9% and 96.3%, and eleven mutants seemed to survive all four suites.

We caught it because all four suites explicitly tested that to_roman(4000) must fail, and yet the mutant that accepted 4000 came out alive. We fixed the runner, checked that case and reran the whole analysis. The numbers in this article are the corrected ones, and the published material includes the fix.

We mention it because it is this article's subject in the first person: a verification that showed green without actually verifying. A common-sense check caught it, not the verifier itself.

What is measured and what is not

Why it may differ from other studies

Leu et al. (ESEM, 2026) found that tests from a different model catch no more than tests from the same one, and that about 40% of defects were caught only by human-written tests. This experiment does not contradict them, because it measures something different under different conditions:

• There were no real defects. Common cause shows up mainly when the code is wrong, and here the code was not wrong.
• Small modules and precise specifications. With the specification in hand, there is less room for the model's bias to decide what is correct.
• Both models come from the same family. Independence across different providers is greater than what we measured.
• One run per cell. We do not know how much a suite varies if the same model writes it again.
• Cells A and C could run their tests against the code. That is part of what seeing the code means in practice, but we did not separate it.

What it means in practice

1. Passing tests do not tell you what they are worth. All 765 tests passed, and still each suite missed between 1 and 8 seeded bugs. Mutation testing makes this visible, and it is cheap.
2. A different verifier is still cheap insurance. In the one module where there was a difference, the other model caught what the same model missed twice. The effect is small, but it points the same way as the first article's recommendation.
3. Beware of properties the code makes true by construction. A round trip only proves something if both directions are computed independently. Pair it with expected values written by hand from the specification.
4. Verify the verifier too. Our own runner reported green on what it had not checked. Seeding a known bug and confirming it gets caught is the cheapest way to know whether a check works.

How we would repeat it at a larger scale

• Tasks with real defects: larger modules, vaguer specifications, or problems taken from real repositories.
• Several runs per cell, to measure variability.
• Models from different providers, and human-written suites as a fifth cell.
• Equal numbers of tests per cell, to separate independence from thoroughness.
• Separating "seeing the code" from "being able to run the tests against it."

What this article does not claim

• It does not claim common cause does not exist. It claims that, in this setting and by the criterion we fixed, we did not observe it strongly, and that the setting had no real defects to test it against.
• It does not claim AI-generated tests are enough. Every suite let bugs through.
• It does not claim the author model never makes mistakes. Twelve small modules do not represent a real system.
• It does not change the first article's recommendation to write tests before the code and add an independent verifier.

Material and sources

• Full experiment material: experimento-causa-comun.zip. It includes the specifications, the reference, the generated code, the four suites, the tools (Python standard library only) and the raw data. Three commands reproduce it end to end. The material is documented in Spanish.
• Design and execution: Claude Code (Opus) with Opus and Sonnet subagents, October 2026. The experiment was also designed by an AI, which is why we publish all the material.
• Knight and Leveson, An experimental evaluation of the assumption of independence in multiversion programming, IEEE Transactions on Software Engineering 12(1) (1986).
• Leu et al., ESEM (2026).
• Dixi-Project: Verification debt, Scope, specify, audit and Teaching software engineering when AI writes the code (2026).