A manual test case in the DI² project had everything the convention asks for: a header with goal and precondition, a step table with step and expectation, an entry in the catalog overview, references to the files it covers. The catalog’s automatic structural check had accepted it. On the first execution it turned out that the path it describes does not exist: the starting state the test case sets up itself makes the only column unusable that a later step builds on.
The next day, a second test case demanded an observation in step 6 that nobody can make on screen. Two weeks later the catalog ran in full for the first time, 30 test cases in five suites, and 16 test case files were changed along the way. Until then nobody had noticed any of these defects, and none of them had failed the automatic structural check.
This article is about what the first execution of a test case actually tests: first the test case itself, then the application, and sometimes the requirement behind it. A test case review, meaning reading the test case against code and database, catches much of this up front. It does not replace the first run, the one QA practice would call a test case dry run. The declarations, meaning the written test cases, and the run logs come from a coding agent, Claude Code with the models Claude Opus 5, Claude Fable 5 and Claude Fable 5.1. The maintainer clicked and judged, including on the requirement.
The essentials up front:
- A test case declaration can be wrong without anyone seeing it while reading: Either the path it describes does not exist, or you cannot see on screen at all whether the expected thing happened. A structural check finds neither defect, because it checks the form and not whether the test case can be carried out.
- The first full run of a catalog tests the test cases first: 16 of 30 test case files were changed during that run in the DI² project. In eleven of those 16 files, it was not the clicking that found the defect but reading the declaration against code and database immediately beforehand. That preparation still did not replace the run.
- A passed run only proves what was actually observed: A “pass” from an earlier run was wrong. Two reports declared several steps passed at once, and a database trace that ran alongside the run did not show all of them. The log belongs at step level, with the trace as evidence.
- Sometimes a passing test case costs the feature it tests: A test case passed, and while clicking through it the decision was made that the tested feature should not exist at all. The test case was retired instead of deleted.
Prerequisite: None. The examples come from a catalog of manual test cases that lives as Markdown files in the repository of a web application. The application manages tables and columns of a database, and that is all you need to know about it. The principles apply to any test case a human executes by following instructions. The text keeps two words apart: test case means a written sequence of steps with expectations that a human executes on screen. Test means an automated check that ends green or red. Automated tests are the subject of the article about the green test that proves nothing, not of this one.
Contents
- What a Declaration Is and Why It Can Be Wrong
- First Failure Mode: The Path Does Not Exist
- Second Failure Mode: The Expectation Cannot Be Observed
- What a Test Case Review Catches and What It Misses
- The Full Run: 16 of 30
- A “Pass” Is a Claim
- When the Test Case Passes and the Feature Falls Anyway
- The Test Case Dry Run in Practice: Five Consequences
- Working Prompts as Examples
- What Holds in the End
- FAQ
- Related Articles
What a Declaration Is and Why It Can Be Wrong
A test case in the DI² project is a Markdown file with a fixed layout: a header with goal and precondition, meaning the state of the database before the first click, then a step table with one step per row and the expected result next to it, and the cleanup at the end. Every run gets its own log, which names the test case by its number and records a result. How such a catalog is built and where its cases come from is described in the article How to Write Manual Test Cases.
This article calls that file the declaration. It is code-like: it describes a sequence that someone else executes later, and it can be defective without anyone noticing while reading. An automatic structural check guards the catalog on every change. It checks the form and nothing else: is the header complete, does the overview list the test case with the same number and the same title, do the files it refers to exist. Whether step 4 can be executed at all after step 3, and whether at step 6 you can see at all whether the expected thing happened, is something the structural check does not see.
A declaration can be checked against the code, and in the DI² project the agent now does that before every run. It reads the precondition against the database and the steps against the application: does this state exist, and does a step create it? Does this button still exist? That finds a lot, as the full run further down shows. One question it answers only in part: whether you can see on screen at all whether the expected thing happened. If the application hides an element in a certain state, that is in the code. That two messages look the same to the viewer, or that a marker disappears at the moment of the click, usually shows up only once someone has the file next to the browser.
Key point: A declaration is a text that can contain defects the way code can. The catalog’s structural check finds missing parts, not wrong content. Whether the described path exists and whether you can see the expected thing on screen is something you know for sure only after the execution.
First Failure Mode: The Path Does Not Exist
The first case dates from 28 August 2026. The application shows a message when a validation rule no longer fits the column it was created for, for instance because the column now holds text instead of numbers. A test case checks that this message counts as resolved on its own as soon as the cause is removed.
To do that, the setup first creates the cause: select a column, create a rule on it, switch the column to text, the message appears. Then the test case names two paths for removing the cause. Either switch the column back. Or, verbatim: “move the rule to a numeric column or delete it”.
The move is not possible. The test case works with exactly one column, and that is the very column the setup has just switched to text. There is no other numeric column in this state, and fetching one ran into an open bug of the application at that time. On this path the test case could not be completed.
Nobody had noticed that while reading. The declaration read as coherent, the structural check was satisfied. It came up on the first attempt to execute it that same evening. Since then the second path reads only “delete the rule”, and the log calls the finding by its name: a defect in the declaration, not in the product.
Key point: A test case can describe a path that is not passable from its own starting state. No single line is wrong in that case, two lines merely contradict each other. You see it when you walk the path.
Second Failure Mode: The Expectation Cannot Be Observed
The second case is one day younger and less conspicuous, because the path exists. A test case checks that deactivating a table also makes its messages disappear. The setup deliberately produces a message on one table by leaving a required field empty. A second table with its own message serves as a control. Steps 1 to 5 passed: table deactivated, its message gone, the second table’s message unchanged. Step 6 asked to activate the table again and expected: “The old messages do not come back.”
On screen, step 6 looked like a failure: after the activation a message was there again, with the same text. Had the old one come back after all? The database said no. The old message had been deleted on deactivation. The visible one was a new record, created four seconds after the activation. The reason: the empty required field that had triggered the message had survived the deactivation. The application found the gap again and reported it again. To the user it looked like the old message, in the system it was a different one. And that was exactly how the application was supposed to behave at the time.
The defect was in the expectation. Old and new message look the same on screen. Step 6 demanded a distinction that nobody can make on screen. The declaration was changed that same noon. Before:
| 6 | Activate the table again. | The old messages do not come back. |
Afterwards the on-screen expectation is limited to what can be checked, and the actual statement is checked in the database:
| 6 | Activate the table again. | If the cause persists, a message with the same
content appears again, as open. All that can be checked on screen: it comes back
as open, not as a resolved leftover. |
> Step 6 needs a counter-check in the database: whether the message is the old one
> (failure) or a newly created one (correct) cannot be told apart on screen. The
> message's ID and creation time must be later than the time of the activation.
> In the reference run: activated 10:24:46, new message 10:24:50, the old ones
> deleted.
This version lasted five hours. That same evening the application learned to clear the table’s settings as well when deactivating. Since then the cause disappears along with it, and step 6 expects that no message appears at all, which can be checked on screen again.
Key point: An expectation is only worth something if the person executing can check whether it occurred. Usually that means the screen. Where the screen is not enough, the declaration has to say where to look instead, here in the database.
What a Test Case Review Catches and What It Misses
On 29 August, the same day as the second case, a third test case was waiting for its first run. It shows how far you get without executing.
The backstory: a bug in one of the application’s dialogs had been fixed. It occurred only with columns that had been selected once and deselected again. The agent had secured the fix with twelve automated tests, and a QA pass with eleven checkpoints had accepted it. That QA pass could not click through the browser, because the login was missing in its environment. Instead it wrote down the click path as a manual test case and reported the bug as fixed and ready to close.
The maintainer asked back whether the test case should not run first before the bug is closed. A search across all eight run logs that existed by then showed: the test case was written, listed in the overview and linked from the bug, but never executed. The bug would have been closed on the basis of tests and QA pass, and nobody would have noticed, because nothing asks for it.
Before anyone clicked, the agent checked the test case against the code. The decisive question: do the first steps of the test case actually create the state in which the bug occurs, that is, a column that was selected once and deselected again? The code confirmed it. So the test case was built correctly.
Then it ran for the first time, and the run still found two defects in the declaration. Step 4 expected a confirmation dialog on deselecting. It did not come, and rightly so: the application only asks when deselecting would delete something along with it. The expectation came from documentation that mentions the dialog in general terms, the exact condition was only in the code. Step 6 asked to see a “NEW” marker on the column in the selection field. It exists, but only in the opened field before the click: selecting adopts the column, and the marker disappears at that same moment. The step described an observation in the wrong place.
Key point: A test case review against the code answers whether a state can be created. Whether an expectation is visible on screen, and at which moment, it answers only in part. In this case only the execution showed it. A test case that has never run is therefore a draft, even if it is well built.
The Full Run: 16 of 30
From 13 to 15 September 2026 the catalog ran in full for the first time: 30 active test cases in five suites, one log per suite, in the local development environment. The maintainer clicks. The agent guides, reads along in a database trace that records the application’s counters every two seconds, and writes the log. All 30 test cases ended green, two of them only after a fix.
The run found 19 bugs in the application, each recorded as its own entry in the project’s bug list and fixed by now, from a wrong sort order to a deactivation that also deleted settings of another table. Only two of them were found by the test cases in the intended way, because a step did not produce the expected result. The other 17 surfaced in passing during execution: because something happened during a test case that no step asked about, while setting up the starting state, while cleaning up afterwards, while re-testing a bug that had just been fixed, or because the agent read the code before a test case and saw something.
The number this article is about is a different one: in 16 of the 30 test case files something was changed during the run. The number does not simply work as a defect rate. The practice was three weeks old, and not every change fixes a defect. Sorted by what triggered the first correction, the 16 files fall into four groups:
- The application had changed since the declaration (six files): A later fix had removed a button, blocked an order of steps or added something to a display, and the declaration had not been updated. One test case did two things in an order the application has not allowed since a fix two weeks earlier.
- A precondition that no step created (four files): A later step relied on something that, according to the precondition, was not there, or it checked values that no step had set. Two of these test cases had apparently been written against an environment in which it already was there.
- A step that never fit the application (three files): One step set a filter that matches every row in the starting state. The table would have been empty, and the thing the step was meant to check not visible at all. Another wanted to toggle a filter while a row was selected, and that is exactly when the application hides the filters.
- Catching up and clarifying after the run (three files): A fix during the run changed how a test case cleans up after itself, and two test cases got more precise information on the starting state after the run. That is maintenance, not a defect.
So 13 of the 16 files contained something that could not be executed or checked the way it was written. Six of them had been overtaken by a later fix to the application without anyone updating the declaration: a catalog ages with every fix.
The mechanism is more precise than the punchline in the title suggests. Eleven of the 16 files were corrected for the first time before the maintainer made the first click of the respective test case: while reading the declaration against code and database. Five files got their correction only during execution or afterwards. Three of the eleven corrected in advance still needed a second one after the run, for instance because a fix during the run had changed a displayed number. In one suite both declaration defects were visible only during execution. One step asked to click on something the previous step had just made disappear. And a test case that had never run since it was created described an element that had not been unambiguous since a later change.
On 16 September, while this article was being written, the pattern repeated on a small scale: two test cases declared at noon were corrected that evening while preparing their first run.
Key point: The first full run of a young catalog tests the test cases first. Most of it is found by the preparation, meaning reading the declaration against code and database before the click, here eleven of 16 corrections. What it does not find, only the execution finds.
A “Pass” Is a Claim
The full run showed something else that concerns not the declaration but the log. A test case checks that a view the user has expanded stays expanded after a recalculation. In the run of 8 September this step was marked “pass”. The code had not changed since. While reading the code before the click, the agent noticed that the recalculation does touch that state. A targeted counter-check by the maintainer confirmed it: the view collapsed. The step had never been green.
The expectation has a property that makes it vulnerable: it claims that something does not change after an action. If the view looks plausible after reloading, such an expectation is easily ticked off, because nothing conspicuous happens.
Twice in the same run, a collective “pass” did not cover all steps according to the trace. While re-testing one bug, the declaration required the number of tables to change between two recalculations, and the trace showed no change for those steps. While re-testing another bug, the English round was missing. Both rounds were repeated.
Key point: A “pass” is at first a promise by the person executing. It becomes evidence only with the log at step level and the trace next to it. Since then the DI² project has a rule: if a step that has to change something in the database is missing from the trace, it counts as not run. And expectations of the form “does not change” deserve a second look.
When the Test Case Passes and the Feature Falls Anyway
Back to 29 August and the test case that had never run. Its first run passed, after the two corrections to the declaration. It checked that an existing validation rule can be moved to another column, a path into which a fix, twelve automated tests and eleven checkpoints had gone. At the third click the maintainer put into words what felt off: it feels wrong to change the column of an existing rule. In terms of the domain, a rule on another column is another rule, and the detour, create a new one and delete the old one, costs almost nothing. That same afternoon the capability was removed.
The test case was not deleted but retired. It is still in the catalog, with the status “Retired” and the reasoning in its header: why the path no longer exists, which class of defect remains reachable through creating a new rule, and which successor test case has checked it since. That reasoning is by now the most detailed version of the product decision in the entire repository. The log of the same day refers to the test case, and 24 test cases later the reference still reaches its target. The passed run remains valid: it proves the state of that day, not today’s.
- **Status:** Retired (decision 2026-08-29: the column of an existing rule can no
longer be changed. A rule on another column is another rule in terms of the
domain, and the detour "create a new one, delete the old one" costs almost
nothing. The move tested here is therefore a forbidden path. The underlying
class of defect remains reachable through creating a new rule and is checked
there by the successor test case. The passed run of 2026-08-29 remains valid
as evidence of the state at that time.)
This can be read as a failure of the requirements phase: the question whether this path is needed at all belongs before the implementation. That is true. In the DI² project the test case is still the only place in the process where someone actually performs a user’s steps one after another. Reviews read diffs, QA passes check criteria, specifications describe intentions. None of these formats produces the feeling “I would never do it this way”. For that you have to do it, and in this process the test case is the last opportunity before a user does.
Key point: A passing test case can cost the feature it tests, because only while clicking through it does someone notice that the path should not exist. An earlier click would have saved the fix. Retiring instead of deleting keeps logs and reasoning intact.
The Test Case Dry Run in Practice: Five Consequences
- Run it once as soon as it is clickable: Not only at the end of the chain, when fix, tests and QA pass are already aimed at the path. The later the first run comes, the more work already depends on the path it calls into question. QA practice has a name for this, the test case dry run: a pass whose only job is to check whether the test cases can be executed the way they are written. The DI² project had no separate dry run, the first real run did that work as well.
- Make the test case review a step of its own: Before the run, read the declaration against code and database. Can the precondition be created, can every step be executed, does every button still exist? That finds most defects before anyone clicks.
- Keep declaration and execution apart: Whoever declares writes against the environment in their head. Whoever executes has the real one. That separation is the reason the execution finds anything at all.
- Keep the log at step level, the trace next to it: A collective “pass” is a promise, the log turns it into evidence. A step that has to change something in the database and is missing from the trace counts as not run. Expectations of the form “does not change” deserve a second look.
- Retire instead of delete: A test case whose feature no longer exists stays in place with status and reasoning. Logs refer to it, and the reasoning is read where someone wants to know why a path disappeared.
What is said here applies to declared manual test cases. For automated tests a related lesson holds with a different mechanism: a green test with stand-ins confirms the assumption that sits in the stand-ins. That is the subject of the article The Green Test That Proves Nothing. Both share the core: a check that has never met reality has not checked anything yet.
Working Prompts as Examples
The sessions are not versioned. The two prompts here are reconstructed from the run logs and commit messages and are not quotations. They show what the maintainer demanded, not how the agent implemented it.
Before the full run, the point was to make the preparation a step of its own:
Before the suite runs: read every declaration against the current code and the
database. Can the precondition be created, can every step be executed, can every
expectation really be seen on screen? Acceptance: per test case and step yes or
no with a reason. Commit corrections to the declaration before the first click,
so that the log names the version that was run.
The agent committed the corrections with reasons and recorded in the log which correction came before which test case. Whether every expectation can really be seen on screen was something it could answer only in part, and that is exactly where the two defects lay that became visible only during execution.
During the run, the point was what counts as evidence:
Log per step what you saw in the trace, not my "pass". If a step that has to
change something is missing from the trace, it counts as not run, and the round
is repeated. Acceptance: every "pass" in the log has a timestamp from the trace
next to it.
Twice the agent then rejected a round because the trace did not show the steps. Both prompts state what is to be checked and what the result is accepted against. How the agent reads and what it records is left to the agent.
What Holds in the End
The first execution of a test case tests three things. First the test case: does the path exist, and can you see on screen whether the expected thing happens? As long as that is open, a step says nothing about the application. Then the application, and that is the normal case the test case was written for. And sometimes the requirement: whether the path should exist at all.
| What is tested | What shows up | What remains as evidence |
|---|---|---|
| first the test case | a path that does not exist, or an expectation nobody can check | the corrected declaration |
| then the application | a step without the expected result, and much in passing | the log at step level, with the trace next to it |
| sometimes the requirement | a path that should not exist | the retired test case with its reasoning |
A test case that has never run is a draft. A passed run is evidence for the state it ran against. In between lies the one check that no reading replaces, whether it is called a first run or a test case dry run: someone with the file next to the browser.
FAQ
As much as a draft. The declaration can describe a path that does not exist, or an expectation that nobody can verify on screen, and neither shows while reading. In the DI² project, 16 of 30 test case files were changed during the first full run, 13 of them because something could not be executed or checked the way it was written. A test case is validated after its first run at the earliest, and even then only for the state it ran against.
No, but it is the best first step. Whoever reads the declaration against the current code and the database finds preconditions without a step, steps that have been impossible since a fix, and filters that empty a table. In the full run that accounted for eleven of 16 corrections. Whether you can see on screen what the test case expects can be read from the code only in part. The execution shows it reliably.
As soon as the path is clickable, not only at the end of the chain. QA practice calls such an early pass a test case dry run. In the DI² project a test case stood before the closing of a bug and had never run. Its first run found two defects in the declaration and a product decision that made the entire path superfluous. An earlier click would have saved the fix.
Retire it, do not delete it. The test case stays in the catalog with status and reasoning, because logs refer to its number and because the reasoning is read where someone wants to know why a path disappeared. Its last passed run remains valid as evidence of the state at that time. If the class of defect remains reachable through another path, it gets a successor test case.
Only for what was actually observed. A “pass” from an earlier run was wrong in the DI² project, because an expectation of the form “does not change” looked plausible, and two collective reports did not cover all steps according to the database trace. Evidence is the log at step level with the trace next to it.
Related Articles
Parent articles:
- AI-Assisted SQL Development with Claude Code — the hub of the AI coding cluster: rules, skills and agents at a glance.
- How to Write Manual Test Cases — the hub of this branch: structure, template and four rules for a catalog of manual test cases.
Sibling:
- The Green Test That Proves Nothing — the automated side of the same lesson: three routes on which a green test suite confirms the bug instead of finding it.
Control and diagnosis:
- The Control Loop — the station “declare test cases” in the recurring control, and the test case that has never run as its limit.
- The Agent Measures Where I Click — the same division of labor in debugging: the human clicks, the agent measures, and a finding only holds with a measurement behind it.