The Green Test That Proves Nothing — Why a Passing Test Suite Confirmed the Bug It Was Supposed to Catch

Published Sep 16, 2026
by

A bug on the DI² project’s website counted as fixed for four weeks. When the reader switches the language, the page is supposed to stay at the spot where the reader was. Three fix commits had promised that one after another, each with its own diagnosis, and after each one the automated test suite was green. Then the maintainer ran a manual test case for it for the first time. On every desktop screen, the bug was still there.

The automated test suite had not lied. It had been written by a coding agent, Claude Code with the Claude Fable 5 model. The suite checked exactly what this agent took to be reality, and the agent had been wrong. The automated test pinned down a value that never existed at this spot in a real browser. It confirmed the behavior under precisely the assumption that was wrong in the real browser.

This article tells that case and two more from the same project and the same week. They reach the same result by three different routes, and each of them gets an article of its own on this blog with all the details. Here they stand side by side so that the shared mechanism becomes visible. The tests, stubs and test data in all three cases were written by Claude Code, with the models Claude Fable 5 and Claude Opus 5. The wrong assumptions this article is about sat in exactly these artifacts. The maintainer initiated and ran the manual test cases.

The essentials up front:

  • A green automated test that runs against stand-ins shows that the code behaves toward those stand-ins the way the test expects: Whether the stand-ins match reality is something it cannot check. If they feed it something wrong, the test is green and the program is broken anyway.
  • Three routes lead an automated test past the bug: The stub, the stand-in for browser or server, plays an environment that does not exist in that form. The right building block is checked, but not at the spot where it is called. Or test and counter-check share the same wrong test data.
  • The counter-check against the old state costs minutes and has a limit: A new automated test is run once against the broken state. If it is red there, it reacts to the code. Whether it matches reality, that run does not say.
  • All three cases were found by a human at the screen: twice with a written manual test case, once with a single click that triggered a guard, a blocking condition in the code, which always fired — the suite as it stood at the time had reported nothing in any of the three cases.

Prerequisite: None. The examples come from a React application tested with Vitest. The principles apply to any test environment in which parts of reality are replaced by stand-ins. The terms stub, fixture and counter-check are explained on first use. The text keeps two words apart: test means an automated check that ends green or red. Test case means a written sequence of steps with expectations that a human runs at the screen.

Contents

The Case: Fixed Three Times, Tests Pass, Bug Still There

The DI² project’s website exists in German and in English. Anyone who has scrolled some way down the start page and then switches the language is supposed to land at roughly the same spot on the target page and not back at the top. That was the requirement, and it sounds like an afternoon’s work.

Three fixes followed. The first, at the end of July, remembered the section anchor the reader had last jumped to. The second, on the same morning, remembered the scroll position of the window instead. The third, four weeks later, fixed a genuine side issue that occurred only in the development environment, because React’s Strict Mode runs effects twice there. Each of the three commits named a cause, and each was marked as done.

On 28 August 2026, a manual test case was written down and run for this bug for the first time: open the start page, jump to a section, switch the language, look at the position. The run failed: The position was not taken into account on the language switch, exactly as on the first day.

The cause was in none of the three diagnoses. On the website, from a window width of 768 pixels upwards, it is not the browser window that scrolls but an inner container, because the layout turns off scrolling of the page body. From that width on, so on every desktop, the scroll position of the browser window was a constant zero. What got remembered was always zero, and the restore moved an element that does not scroll at all. Below 768 pixels the window scrolls normally. There the fix worked, and that fed the misdiagnoses: Anyone who looked in a narrow window or on a phone saw a working feature.

Neither the maintainer nor the agent had noticed the bug when trying it out in the browser, three times over. That is one half of the story. The other half is the automated test suite, which was green on every run during those four weeks.

First Route: A Stub with the Wrong Contract

The function that remembers the scroll position had an automated test in Vitest. It ran in the Node environment without a simulated browser, and there is no window and no scroll position there. So the test gets a stand-in object, a so-called stub, that plays the role of the window. The test gave this stub a scroll position of 842 pixels and, along with it, a stand-in for the browser’s session storage that remembers values only for the duration of the test. Then it called the function and checked whether the rounded value 843 had landed in that storage.

  1: it("remembers the rounded scroll position", () => {
  2:   const sessionStorage = fakeSessionStorage()
  3:   vi.stubGlobal("window", { scrollY: 842.6, sessionStorage })
  4:
  5:   rememberScroll()
  6:   expect(sessionStorage.getItem("lang-scroll")).toBe("843")
  7: })

The test was correct in its environment. The function read the window’s scroll position, rounded it and stored it. That in a real browser on a desktop it always read zero was something the test could not know: The stub had given the window a scroll position it never has there. The test confirmed the behavior under the assumption that the window scrolls, and precisely that assumption was wrong on the desktop.

One cannot blame the stub for that. The mistake lies one step earlier: Nobody had checked whether what the stub plays matches what the browser actually does. A stub is a written-down assumption about the environment, the contract the code believes it has with it: The window scrolls, and its position can be read. Assumptions one does not recognize as such, one does not check.

The fourth fix gave the inner container a fixed ID, read the position there first and fell back to the window only when the container does not scroll. The new test stubs both: a window that stands at zero and a container that stands at 640 pixels.

  1: it("reads the position from the scroll container when that is what scrolls", () => {
  2:   const sessionStorage = fakeSessionStorage()
  3:   vi.stubGlobal("window", { scrollY: 0, sessionStorage }) // window stands still (desktop)
  4:   stubScrollHost({ scrollTop: 640 })
  5:
  6:   expect(readScrollTop()).toBe(640)
  7:   rememberScroll()
  8:   expect(sessionStorage.getItem("lang-scroll")).toBe("640")
  9: })

This test is better because its stub reproduces the situation the bug lives in: a window that stands still and a container that scrolls. On the old code it fails with the observed symptom, expected 640, read 0. The four attempts in detail and a second case of the same class two days later have an article of their own.

Second Route: The Function Tested, the Call Site Not

The second case is one day later, in the dialog in which the validation rules for a table are created. Its column field also offers columns that are not yet selected in the project, marked “NEW”. If the user picks such a column, the application is supposed to select it in the project right away, a write to the server. It did not do that when the column had been selected once before and deselected again. The dialog did not decide on the column’s state but on an internal value: Columns that have been captured once carry an ID, all others their name. A deselected column keeps its ID, so the dialog took it to be selected, skipped the call, and the server rejected the rule.

The fix pulled the decision into a pure function that looks only at the column’s state, with twelve automated tests. All twelve were green, and so was the whole suite with its 773 tests. The QA pass afterwards asked what a green run does not answer: Would the twelve tests report the bug if it came back? The QA agent temporarily put the old decision back into the dialog. All twelve tests stayed green, because they call the function directly, and the function still answers correctly. Whether the dialog uses it at all is something they do not see. The only thing that turned red was a new test that renders the dialog, clicks a deselected column and checks whether the call is sent.

Here no bug survived four weeks. The gap was found by deliberately asking about the old state. The case still belongs in this series because it shows a second route: A fully tested building block says nothing about whether it is wired in. In the project this has become an error class of its own. Five later test suites cite this case as the model for securing the call site specifically, as of 16 September 2026. The QA pass, the tests and screenshots from the application have an article of their own.

Third Route: A Wrong Fixture in Test and Counter-Check

The third case is on the same afternoon as the second and is the most uncomfortable one, because it hits the countermeasure that had come out of the first case: The test contained a server response the real server never delivers, and the counter-check ran against the same response.

The same dialog got a new feature: Anyone who picks a column marked “NEW” there and then discards the dialog is supposed to get the column deselected again. Because deselecting would also reset dependent validation rules, foreign keys and key columns that belong together, the dialog first asks the server what hangs on the column and leaves it in place if the response contains anything. The response contains a list of column IDs. The agent read it as a list of only the other columns that would be reset along with it. In fact it contains all affected columns, so always the requested one as well. The condition “list not empty, so block”, a so-called guard in the code, was therefore always true: The rollback never ran, and the notice “rules now depend on this” appeared on every discard.

The component tests were green, type check and linter clean. And so was the counter-check: The agent had unhooked the rollback, the tests for it turned red, the rollback came back, the tests turned green again. By the rule from the first case, everything had been done. The cause sat in the fixture, meaning the prepared server response that the test slips to the dialog. It delivered an empty list, which the real server never delivers for a requested column. Test and counter-check shared the same mistake. What found it was the first real click: The maintainer ran the manual test case for discarding, saw the column stay in place and the notice appear, and reported both. Since then the fixture echoes the requested ID back, and against this version the old guard turns red.

  1: // fixture before: a response the server never gives to this request
  2: impact: { cascadeColumnIds: [], rules: [], fks: [] }
  3:
  4: // fixture after: the requested column is always included
  5: impact: { cascadeColumnIds: [Number(input.columnMetadataId)], rules: [], fks: [] }

One circumstance makes this case more than an anecdote: The sentence “A test that stubs its own assumption only confirms itself” had been in the commit message for the scroll fix since the afternoon of the previous day, written by a Claude Code session with Claude Opus 5. A good 24 hours later, another session in the same project, this time with Claude Fable 5, made the same mistake in a different form, although rule and contract were in the repository. Whoever writes a fixture writes down an assumption about the server. And whoever does not recognize an assumption as an assumption does not check it, no matter how many rules they know. How the rollback came about, the comment in the fixture and two more cases of the same class have an article of their own.

The Counter-Check: Every Regression Test Once Against the Old State

Out of the first case a way of working has emerged that can be read off the project’s commits. In the six weeks before 28 August 2026, none of a good hundred fix commits mentioned a counter-check against the old state. In the two weeks after, almost every second one did. The way of working goes like this: A new automated test that is meant to secure a fixed bug is run once against the old, broken state. For that, the fix is temporarily reverted or unhooked, the tests run, and they have to fail there with the observed symptom. Only then is it shown that they react to this bug, and only then does the fix come back.

The cost is small: a temporary revert and a test run, a few minutes in a project of this size. That holds as long as the old state can be restored with little effort. If the fix is part of a larger rebuild or the bug occurs only under load, the counter-check becomes more laborious, but not superfluous. The benefit is the answer to the question a green run alone does not answer: Would this test have found the bug? In the first case the answer was no for the old test and yes for the new one, in the second no for twelve tests and yes for one.

For someone who cannot read the code, this rule is especially valuable. It produces a result that can be judged without knowing the language: two runs, one red, one green. The maintainer of the DI² project knows neither TypeScript nor the test library. He can still demand the counter-check and sign off on its result. That is exactly how it came into the project.

The third case shows the limit. The counter-check proves that a test reacts to the code. Whether the fixture matches reality is something it cannot say, because it runs with the same fixture. The addition costs hardly more: The fixture is compared once with what the system actually delivers. Sources for that are the interface documentation, the server code or a real response from the running application. What gets written down is the property the test relies on, not the whole response. In the third case one sentence would have been enough: The list always contains the requested ID. A comment on the fixture does age with the system, though. Anyone who wants to secure the property permanently additionally writes an integration test that checks exactly this property against the running application.

Two checks, both small, both with an unambiguous result:

  • Against the old state: Is the new test red there? If not, it does not check what it claims to check.
  • Against the real contract: Does the system behave the way the fixture claims? If not, test and counter-check together check the wrong world.

What This Means for the Manual Catalog

The three cases are not a plea against automated tests. Within these two days the project’s suite grew from 761 to 804 tests, and the regression tests from all three cases are part of it. Next time, they will report the three bugs in seconds without anyone having to click.

What the three cases show is something else. In all three, the only check that looked at the screen in a defined state was a human with a manual test case. The test case for the language switch consisted of seven steps with one expectation each and demanded a viewport wider than 1,024 pixels. Its first run refuted a fix that had counted as done for four weeks.

What has become of this experience as a way of working in the DI² project is described in the article on the control loop. How a manual test case is built, where the cases of a catalog come from and what a run proves is the subject of an article of its own on this blog. And that the first run of a test case first tests the test case itself before it tests the application is a second one.

Working Prompts as Examples

The sessions in which the three cases were handled are not versioned. The two prompts here are reconstructed from the bug files and commit messages and are not quotations. They show what the maintainer demanded, not how the agent implemented it.

After the fourth fix of the scroll bug, the point was to trust the new tests without being able to read them:

Temporarily revert the fix and run the three new tests.
Acceptance: They are red, and the message shows the symptom from the test case
(0 is remembered instead of the real position). Then put the fix back in and
document both runs in the commit.

The agent ran both runs and copied the message of the red run into the commit message. That message is the proof, not the green run after it.

After the manual test case failed in the third case, the point was to clear up the contradiction between a green suite and a red click instead of just repairing the guard:

The manual test case for discarding has failed: The column stays selected,
and the notice appears on every pass. The tests and the counter-check were
green. First clarify why they did not report that, then correct the guard.
Acceptance: The fixture reflects what the server really delivers, the old
guard turns red against this fixture, and the reason is written as a comment
on the fixture.

The agent found the cause in the contract of the response, brought the fixture in line with the server’s behavior and ran the old guard against it: two tests red. The comment that has been on the fixture ever since names the property the first attempt had overlooked: With the real server, the list always contains the requested column itself.

Both prompts state what is to be checked and what the result is accepted against. How the agent implements that, with which library functions and in which file, is left to the agent. The maintainer could not prescribe it.

What Holds in the End

The three cases differ technically and have the same cause: Each check was consistent within its own assumptions, and nobody had checked the assumptions themselves.

RouteWhat the test checkedWhat was missingFound by
Stubthe code against a window that never scrolls like thatcomparing the stub with the real browsera manual test case
Call sitethe function, not its call in the dialoga test where the bug livesasking about the old state
Fixturethe code against a response the server never givescomparing the fixture with the real servera manual test case

Five questions a green run does not answer and that someone therefore has to ask:

  • Does the stand-in match reality? What the stub plays has to match what browser or server actually do.
  • Is the building block actually wired in? A test at the call site checks whether the tested function is called at all.
  • Does the system deliver what the fixture claims? Check it once against documentation, server code or a real response, and write down the property the test relies on.
  • Does the test turn red on the old state? If not, it does not check what it claims to check. The counter-check costs minutes and can be signed off without knowing the language.
  • Is there a check that does not share the test’s assumptions? In all three cases that was a human with a manual test case. That does not replace a suite. It checks what the suite cannot see.

FAQ

Why do the tests pass but the bug is still there?

Because an automated test can only check what its environment plays for it. If it replaces browser, server or database with a stand-in, it checks the code’s behavior toward that stand-in. If the stand-in plays something that does not exist in reality, the test is green and the bug stays. The question then is not “is the test right?” but “is the assumption in the stand-in right?”.

Are stubs and mocks the problem?

No. In a test environment without a browser, something has to take its role, and in a component test nobody wants a real server. The mistake is not in replacing but in the missing comparison: Nobody checked whether the stand-in does what the real system does. That comparison is a look into the interface documentation or a call against the running system, compared with what the fixture claims.

How do you check whether a regression test would find the bug at all?

By running it once against the broken state. The fix is temporarily reverted, the test runs and has to be red, then the fix comes back. The result is two runs, one red, one green, and both can be judged without knowing the language. If the test stays green on the old state, it does not check what it claims to check.

Is the counter-check against the old state enough?

Not against a mistake in the fixture. The counter-check runs with the same fixture as the test. If the fixture is wrong, both are wrong, and the counter-check still reports red and green as expected. It proves that the test reacts to the code. Whether the fixture matches reality is checked only by a comparison with the system’s real contract.

How do you check AI-generated tests for wrong assumptions?

The error class is older than any coding agent. A stub that pins down its author’s own assumption is a classic mistake from handwork. In the AI context, though, the second perspective is missing: The same instance writes fix, test, stub and fixture and carries its assumption into all four. Where no second developer questions the fixture, checks that do not share the assumption have to take that developer’s role: once against the old state, once against the real contract.

Parent articles:

Control and diagnosis:

  • The Control Loop — how the counter-check against the old state became part of a recurring control when you cannot read the code.
  • The Agent Measures Where I Click — the opposite direction: why a red test run in a shared working tree is at first only a hypothesis.