Writing Manual Test Cases Before the Code — How the Origin of a Catalog Flipped in Seven Weeks and Which Eight Bugs It Found

In mid-September 2026, this blog published a number about the catalog of manual test cases in the DI² project: 27 of 32 test cases existed only because something had been broken before. Whether that share would fall as the practice matured was left open. A third data point was supposed to show it, a few months out at the earliest.

Seven weeks later the catalog has 159 active test cases, and 31 of them are bugfix-derived. The share has fallen from 84 to 19.5 percent, and the reason was not a matured practice, because the rules for writing test cases have stayed the same. What changed was the point in time at which a test case comes into being, and at the same time a phase with many features arrived. Bugs kept coming, in these seven weeks four more bugfix-derived test cases were added, from 27 to 31. The share fell because the test cases derived from requirements grew many times faster.

This article tells how the decision came about to write test cases before coding, shows the share of bugfix-derived test cases across the ten weeks since the catalog started, and takes stock of what a catalog that came into being this way found in 34 runs, what it did not deliver, and why the more important findings sit with the test cases that passed.

All numbers come from one project, one maintainer and ten weeks of catalog. They show what happened in this project. Whether writing test cases before coding works the same way in other projects cannot be derived from them.

The catalog, the test cases since this decision, the automatic structural check that verifies on every change whether a test case is built correctly, and the run logs were written by a coding agent, Claude Code mostly with the model Claude Opus 5.5, alongside Claude Fable 5.1 and Claude Opus 5, following the maintainer’s instructions. The maintainer made the decision, he clicks through the runs and reports what he sees. He does not read the application’s code himself.

The essentials up front:

• The share of bugfix-derived test cases fell from its peak of 90 percent in early September to 19.5 percent because the point at which test cases are written moved and a phase with many features followed, not because the test cases were written better: Since mid-September the test cases come into being in the UX step, from the acceptance criteria of the requirements, and the build steps wait for them. In seven weeks, 123 test cases derived from requirements were added this way, and four derived from bugs.
• The test cases have to exist before the code, and the only way to enforce that is to make the build wait: The agent’s first proposal was to write the test cases in QA, after the build. A test case written then describes what the program already does, and nobody can tell from a finished test case when it was written. That is why in the DI² project backend and frontend do not start as long as a click-only criterion has no test case.
• In 34 runs, test cases failed ten times. Every time the bug was in the feature just built, not once in the test case itself: Most of these test cases had been written before the code existed. One revealed that an entire search function had failed. Another found, in the copy of a project, rules of the application pointing at the wrong columns, although the feature’s automated test had considered them correct, because it had only compared them word for word with the original.
• Defects in the test cases themselves and changed requirements are not found among the failed test cases but among the ones that passed: A test case that is corrected during a run or adapted to a changed requirement runs once more afterwards and appears in the log as passed, with the correction only as a note beside it. Around a dozen test cases were corrected this way, for five features the first look at the UI changed the requirement, and twice a “pass” was wrong, meaning a step was reported as passed that was not. And since the catalog had 30 test cases, it has not run in full a second time.

Prerequisite: None. The structure of a test case and the rules for writing one are described in the article How to Write Manual Test Cases, this article is about what happened afterwards. The examples come from a web application that manages the tables and columns of a database. It is built in a fixed workflow that a coding agent works through step by step: requirement with acceptance criteria, UX step with the mockups, build of backend and frontend, QA, review, release. Wherever the text names one of these steps, this workflow is meant. The subject is manual test cases that a person executes on screen, not the automated tests that run by themselves on every change. For those there is the article The Green Test That Proves Nothing. One word from this application comes up several times and does not belong to the world of testing: a validation rule is a rule with which the application checks the data in a column, for example against an allowed range of values. It has nothing to do with testing the application.

Contents

• The State in September and the Open Question
• Ten Click-Only Criteria, Zero Test Cases
• The Gate in the Wrong Place
• The Bugfix-Derived Share: From 90 to 20 Percent
• What the Test Cases Found
• What Else Happened in the Runs
• What the Catalog Did Not Deliver: The Regression Test
• What the Timing Delivers and What It Does Not
• Working Prompts as Examples
• What Holds in the End
• FAQ
• Related Articles

The State in September and the Open Question

A test case in the DI² catalog has two possible origins. It comes from a requirement and checks whether the application meets an acceptance criterion. Or it comes from a bug and checks whether the bug returns. The catalog overview keeps a line of its own with the number of active test cases that are bugfix-derived. The structural check recalculates this line on every change, because it follows from the field in which every test case names its requirement or its bug.

In mid-September this line stood at 27 of 32. The hub article was careful in how it read that: the share says something about the composition of a catalog, not about its quality, and that a three-week-old catalog consisted mostly of bugfix-derived test cases was to be expected. Whether the share would fall once more test cases came from requirements was for a later data point to show. With this article that data point is here, and it turned out differently from what the careful reading suggested.

Key point: The hub left open whether a catalog gets more test cases from requirements by itself over time. The answer after seven weeks: not by itself. What turned it around was a decision and the phase with many features that followed it: since then the test cases are written in the UX step, before anything is built, and every new feature brought its own.

Ten Click-Only Criteria, Zero Test Cases

One day after this count, a feature was finished. Its requirement had 19 criteria, ten of them explicitly marked as ones that can only be checked on screen. The QA step, in which the agent signs off each criterion after the build against the code and through automated tests, had confirmed all 19. Then the maintainer asked whether there were test cases for this feature. There were none.

That was not carelessness. The workflow had two gaps. First, nobody was responsible for writing the test cases. The tool used in the project to write requirements marked the click-only criteria correctly and noted that the test cases would come “later”, after the frontend was built, without saying who writes them. The four steps after that, namely build, QA, review and release, did not mention the catalog with a single word. Second, nothing stopped the workflow when test cases were missing. The structural check checked the catalog against itself, that is fields, names and agreement with the overview. It did not know the requirements. Ten click-only criteria without a test case were green for every check in the project.

A measurement across all requirements with click-only criteria found exactly one with a backlog, namely this one. The backlog was small but invisible, and nothing in the workflow would have asked about it. An addendum from the same session fits in here: the marker for click-only criteria existed in two spellings, and a counter that knew only one of them would have reported nine instead of ten and stayed green. So the marker was first fixed to one form, and only then does anything count it.

Key point: A duty that does not say who fulfils it is fulfilled by nobody, and a check that knows only the catalog cannot see what is missing from the catalog. The backlog came to light through a question.

The Gate in the Wrong Place

The real finding came in the conversation about the repair. The first proposal came from the agent and was that the QA step writes the test cases. That sounded plausible, because the QA step goes through every criterion one by one anyway, and the click-only criteria, which it cannot check itself, are exactly the ones that need a manual test case. If the structural check had additionally looked up for every criterion whether a test case exists, it would even have been measurable.

The maintainer’s objection overturned the proposal. The QA step runs after the code. A test case that comes into being there is written from the finished program and not from the criterion. That removes the only separation a one-person project with a coding agent can create: the separation in time. So the first proposal would not have meant no gate at all, but a gate at the wrong time. It would have ensured that a test case file exists for every click-only criterion. Whether the expectation in it comes from the criterion or was copied from the finished program cannot be seen from the file, and the catalog counts files. A solution like that feels right and looks right in every overview, and that is exactly what makes it the more dangerous one.

The decision from the same day has four parts. The author of the test cases is the UX step, in which the mockups are created. It is the first step that knows the click path as well as the criterion, and the last one before the build. The gate is the two build steps for backend and frontend. They do not start as long as the requirement has a click-only criterion without a test case. The QA step no longer writes test cases itself. It goes through the existing ones and notes in the log which ones it cannot execute on screen because it lacks a browser or a login. The maintainer clicks through those afterwards. The structural check, which until then knew only the catalog, has since also read the requirements and looks up for every click-only criterion whether a test case exists. It is the safety net behind the gate: it runs on every push and keeps a backlog visible in case the gate is bypassed after all. Whether the steps and expectation of a test case actually match its criterion is something it still does not check. For bugs, the rule is deliberately narrower. A bugfix-derived test case is created only if the bug can be reproduced on screen alone and its fix cannot be secured with an automated test. A gate that fires on every bug would force more test cases than a person would ever execute.

Key point: The build has to wait, not QA. The first step that knows the click path as well as the criterion is the UX step, and that is where the test cases have been written since.

The Bugfix-Derived Share: From 90 to 20 Percent

The catalog overview lives in the repository, and every change to it is a commit. From the history it is therefore possible to read for every day how many test cases were active and how many of them were bugfix-derived:

The same eleven data points as a line, with the day of the decision as a vertical marker:

Numerator and denominator read separately: the bugfix-derived test cases grew from 27 to 31 in seven weeks, the ones derived from requirements from 5 to 128. So the share fell almost entirely because the denominator grew. In the second half of October alone, 46 test cases were added, and not one was bugfix-derived.

That the numerator almost stood still has two causes, and both follow from the decision of mid-September. One is the narrow rule for bugs. The other is less obvious: a bug that shows up in a feature just built does not get a bug file of its own in the project. It is fixed during the run and noted in the requirement. But the overview counts a test case as bugfix-derived only if it refers to a bug file. All eight bugs the catalog has since found at its expectations therefore produced no bugfix-derived test case. They already had a test case derived from a requirement, and that one had found them.

The denominator did not grow evenly. The 123 new test cases came with 20 commits of the UX step, one per feature, with two to 13 test cases each. Counted by month, 18 test case files were created in August, 40 in September, 66 in October and 36 in the first days of November. The decline of the share therefore has two causes that cannot be separated: a gate and a phase with many features. Without the gate, this phase might have produced features without test cases like the one on the day of the decision, and without the phase the gate would have had nothing to enforce. For eleven features the order can be proven from the commit times: first the UX step with mockup and test cases, then backend, then frontend, 18 minutes apart for the smallest feature, three days for the largest.

Key point: The share measures the composition of the catalog, not the quality of the test cases. It fell because every feature got its test cases before the build, not because less went wrong.

What the Test Cases Found

Since the decision there have been 34 run logs. Each records in one line per executed test case whether it passed, failed or was skipped, 166 lines in total: 136 passed, ten failed, 19 skipped, and one line was left open because a visual check was not reported back individually. Behind the ten failures are eight bugs in the application, because twice the same cause hit two test cases. All eight were in the code of the feature just built and were fixed, most of them in the same run, none later than two days after. None of the ten failures was due to a defect in the test case, none to the environment.

Six of the eight were found by test cases written before the build. An example from October: a new tab for an activity log got a search, and the test case demanded in step 8 that searching for a table name shows only the matching entries. Instead, every search ended in an error banner. The database query contained a syntax error, and in the DI² project no automated test runs the text of a query against a real database. The function had failed completely, and no test noticed, only the person at the screen.

An example from November shows how a manual test case and an automated test pass each other by. A feature copies an entire project including its validation rules, that is the rules with which the application checks the data in a column. The test case demanded that the validation rules of the copy point at the columns of the copy. They pointed at the columns of the original, and the copy reported “removed column”. The automated test of the copy operation had compared the rules word for word with the original, and word for word they were identical. It took the bug for correct. That is the pattern from the article about the green test that proves nothing, only that here the manual test case found the bug.

The third example out of these six was seen by nobody on screen. A test case demanded that a connection test leaves exactly one entry in the activity log. The agent conducting the run read the state of the database in parallel and found two entries in the same second. The cause was in the feature’s code: two places wrote the same entry, once the connection test itself and once the saving of its result. The second place has not written an entry since, and the next connection test left exactly one. The maintainer had seen nothing wrong on screen. The bug was found by the database trace that the article about the first run describes as evidence.

The two remaining bugs come from the run on the day of the decision, from the feature that had triggered the decision. Its test cases were written after the code, but from the criteria. In step 5 of two test cases a warning marker that the criterion demanded was missing on screen: drawn, but in the color of its background and half covered by the neighboring cell. The automated test had checked that the marker is in the document. Whether it is visible is something such a test cannot check.

Key point: A test case whose expectation was fixed before the code can fail against the first code, and in the DI² project it did so six times. Two more bugs were found by test cases written only after the code, but likewise from the criterion. All eight bugs were in the feature just built, and no automated test had seen them.

What Else Happened in the Runs

Whoever counts only the failures sees a catalog whose test cases were never wrong. That is down to the convention: a test case corrected during a run runs again afterwards, the line ends with passed, and the correction stands beside it as a note. The 34 logs therefore hold four kinds of events among the passed test cases, and none of them shows up in a count of the failures.

The test case was wrong: Around a dozen test cases were corrected during a run, for instance because a test case expected a confirmation dialog that the UI does not show at all in that situation. These are the same failure modes as with the first 30 test cases in September, only noticeably rarer. That fits test cases whose steps come from a mockup instead of from memory.

The requirement was wrong: For five features the first run changed not the code but the criterion. The clearest case: a test case passed its first step, a cell without content stayed empty, as the expectation demanded. At the sight of the empty column the maintainer wanted a placeholder in every row, and the expectation was changed after it had passed. In another case a criterion demanded a behavior that an older rule of the application deliberately prevents, and the criterion was withdrawn.

The bug was next to the step: Around 25 bugs came to light during the runs without any step demanding them, 18 of them were old and lay outside the feature being built. Against eight bugs at the expectations, that is a ratio of a good three to one. In the first full run in September it was 17 against two. The ways of counting differ, so a comparison in percent would not be serious. The direction is still recognizable: the new test cases find more at their steps than the old ones, and most is still found by the person who looks while clicking.

The “pass” was wrong: Twice the maintainer reported steps as passed that were not. Once an element had only appeared at the next change of view, once five steps had run on a different record than the one the test case demanded. Both times only the database trace showed that the “pass” was wrong.

Key point: The ten failures show only the bugs a test case found at its expectation. What was wrong with the test case itself, where the requirement was changed at the first look at the UI, and where a “pass” was wrong, sits with the passed test cases. Whoever counts only failures sees half of their runs.

What the Catalog Did Not Deliver: The Regression Test

Every test case carries a run class. A smoke test case runs on every release. A regression test case runs when one of the files it covers has been changed, and additionally before every release to production and once a quarter across the whole catalog. So says the rule for the run class. Nothing enforces it, and since mid-September it has not been followed. Nor was that necessary so far. The rule protects users from bugs that return with a later change, and in a private project there are no such users.

The first full run took place in mid-September, over three days, with 30 test cases. Since then the catalog has grown fivefold, and there has been no second full run. The 34 runs since are acceptance runs per feature and repetitions per bug. The catalog has 160 test case files, 159 active and one retired. 158 of the 159 active test cases have run at least once, most of them exactly once, at the acceptance of their feature. Against production, a single log with a single step was run. The test case that has never run is bugfix-derived and needs the server environment, two accounts and two browsers.

The material for this article already held a suspicion about this in September, and the runs since confirm it: a due rule that fires constantly during a build phase is not followed, not out of carelessness, but because it demands more than the day allows. The first full run was not triggered by a changed file but by an announcement: everything, once. For a catalog with 159 test cases, several suites of which need two accounts, a dedicated test project or the server environment, that is the realistic way. A full run is then a milestone with a date.

What the catalog did deliver in this time is something else. All 19 cases in which a test case was skipped name their reason. 14 come from the agent’s QA step, which lacks a browser, a login or accounts, and in every case the maintainer executed the test cases on the server the same or the next day. A skipped test case is thus a visible backlog, not a forgotten one.

Key point: The run class promises a regression run that has not happened since mid-September. Most of the 159 test cases have run exactly once. Whether they hold up as a regression net is therefore untested. At this size, a full run needs a date, not a trigger.

What the Timing Delivers and What It Does Not

In the DI² project the same instance writes the requirement with its criteria, the test case and the code. The decision of mid-September changed nothing about that, it only fixed the order. The runs since show fairly precisely what this order delivers and what it does not.

It delivers that the expectation is not copied from the code. Six bugs at test cases written before the build prove this, because an expectation copied from the finished code adopts its bugs instead of finding them. Had the test case described the finished code, it would have taken the failed search and the wrongly linked validation rules as given. The two bugs from the day of the decision show alongside that an expectation written from the criterion after the code still finds things. What matters is the source of the expectation, the timing secures the source.

It does not deliver that the expectation is right. A test case written before the code from an incomplete criterion passes against code that fulfils the incomplete criterion. The five requirements changed at the first run are the proof, and the case with the placeholder is the clearest: the expectation had passed, and it still was not what the maintainer wanted. Whoever writes the criterion and at the same time knows how they will implement it writes it, in case of doubt, the way they will implement it. The error in the criterion is seen only by a person at the finished UI, and the run is the moment when that happens.

Key point: The timing before the code secures the source of the expectation, not its correctness. Six findings prove the one, five changed requirements the other. In a one-person project the order takes over exactly one thing from the second person who would otherwise write the test case: the test case does not come into being along the finished code. It takes over nothing more.

Working Prompts as Examples

The sessions are not versioned. The two prompts are reconstructed from the rules of the catalog and the notes on the session in which the decision was made, and they are not quotations. They show what the maintainer demanded, not how the agent implemented it.

The first is the objection to the proposal of making the QA step the author of the test cases. It fixes the what and leaves the how to the agent:

The QA step as author of the test cases is too late. It runs after the code,
and a test case written there is written from the finished program, not from
the criterion. The test cases must be written before the code.

Propose which step in our workflow is the earliest in which both the
criterion and the click path are fixed, and how the build steps enforce
this. Acceptance: a requirement with click-only criteria and no test case
blocks the build, and the check on every push shows the backlog.

The QA step as author of the test cases is too late. It runs after the code,
and a test case written there is written from the finished program, not from
the criterion. The test cases must be written before the code.

Propose which step in our workflow is the earliest in which both the
criterion and the click path are fixed, and how the build steps enforce
this. Acceptance: a requirement with click-only criteria and no test case
blocks the build, and the check on every push shows the backlog.

The UX step as author was the proposal of the agent, which knows the project’s workflow. The condition “before the code” came from the maintainer.

The second is the handover to the UX step, as it has read for every feature since:

Here are the requirement and the approved mockups. For every click-only
criterion, write a test case following the catalog's convention: expectation
from the criterion, steps from the mockup, at least one statement about what
must not happen, step and expectation side by side.

Acceptance: the catalog's structural check is green, and every click-only
criterion of the requirement has a test case.

Here are the requirement and the approved mockups. For every click-only
criterion, write a test case following the catalog's convention: expectation
from the criterion, steps from the mockup, at least one statement about what
must not happen, step and expectation side by side.

Acceptance: the catalog's structural check is green, and every click-only
criterion of the requirement has a test case.

The prompt does not repeat the rules, it refers to them. The acceptance criterion turns a suggestion into a requirement: without the second condition the catalog would be green even if a criterion were missing.

What Holds in the End

The share of bugfix-derived test cases in the DI² catalog fell from 90 to 19.5 percent, and the reasons were a gate before the build and a phase with many features, not a matured practice. The test cases that came into being this way found eight bugs at their expectations in seven weeks, all in the feature just built, and among the passed test cases they showed where the test case was wrong and where the requirement was.

For a catalog of your own, three decisions follow from this:

And in every run it is worth looking at the test cases that passed. The log says beside them whether the test case was corrected, whether the requirement changed at first sight, and whether a “pass” was held against the trace.

A catalog does not become reliable because its test cases passed once. For that they have to run again after later changes, and that is the run that has not happened since mid-September. The timing before the code secures where the expectation comes from. It is not evidence that the expectation is right.

FAQ

Related Articles

Upstream:

• How to Write Manual Test Cases — the hub of this topic: the structure of a test case, the four writing rules and the state of the origin in September that this article picks up from.
• The First Run Tests the Test Case — the first full run of the catalog, the two failure modes of a declaration and the trace as evidence.

Sibling articles:

• The Green Test That Proves Nothing — why an automated test can confirm the bug it was supposed to find, recognized here in a test that compared validation rules word for word.
• Unit Test Passes, Integration Fails — the case in which a manual test case found the bug that twelve green tests did not see.

Parent articles:

• AI-Assisted SQL Development with Claude Code — the hub of the AI coding cluster: rules, skills and agents at a glance.
• Agentic Coding from a User’s Perspective — the experience of a user who does not write the code themselves.

Control and rules:

• The Control Loop — the recurring control of code you cannot read yourself, with the test cases as one of its stations.
• The Agent Measures Where I Click — the same division of labor in debugging, with the database trace as evidence.
• Why sqlfluff Can’t Lint Our SQL Conventions — a dedicated check script as a safety net for rules in code, the same mechanism as here for a duty in the workflow.