Unit Test Passes, Integration Fails — Three Questions You Can Ask Without Reading the Code

Published Sep 20, 2026
by

In the DI² application a validation rule could not be saved. The dialog offered the user a column that had not yet been selected in the project. A click on that column should have selected it. In one particular case it did not, and the server rejected the rule. A coding agent wrote a fix. It moved the check for whether a column first has to be selected out of the dialog into a function of its own and added twelve unit tests for it. All twelve were green, and so was the whole test suite with its 773 tests.

Whether that had fixed the bug was not yet established. Before the bug was closed, an agent checked the fix in a separate QA pass. That raised a question a green run does not answer: Would any of the twelve tests turn red if the bug came back? The answer was no.

The twelve tests call the function directly and check whether it answers correctly. Whether the dialog uses the function at all is something they do not see. But the bug sat in the dialog, at the call site. That is the place in the program where the dialog asks the function and then calls the server or leaves it be. Put in short form: unit test passes, integration fails. More precisely, every building block passes its tests and the application is still faulty, because one building block is not wired in or is wired in wrongly.

The short version of this case is in the overview The Green Test That Proves Nothing, which describes three routes by which a green test leads past the bug. This article tells the second route from the point of view of someone who cannot read the code. It shows what was visible on the screen, which question revealed that no test would notice the bug, and which three questions follow from that for every fix. This does not apply to user interfaces only. In the database, too, a SQL function can be tested while the procedure that is supposed to call it takes a different path, and a section of its own covers that.

The fix, the tests and the QA pass come from Claude Code with the Claude Fable 5 model. The maintainer of the project does not know TypeScript. He found the bug while running a manual test case at the screen, triggered the QA pass and ran the corresponding test case himself before the bug was closed.

The essentials up front:

  • What the twelve tests do not see only shows in a run against the old, faulty state: The QA agent wrote a test for this that operates the dialog like a user, and against the state before the fix that test turned red. The project got such a test permanently 45 minutes after the fix.
  • Whether a test sits in the right place after a fix can be asked without reading the code: Three questions are enough, namely where the bug sat, whether a test turns red when that place is turned back to the old state, and whether a missing test is intent or oversight. The answers are one sentence and two test runs, and both can be judged by someone who does not know the language.
  • In the project the case has become a model: Six later test suites check, as it does, whether a tested building block is also wired in correctly, and for the connection between a building block and the place that uses it the project has a word of its own: wiring. Still, not every connection gets a test. Where the project deliberately goes without one, it enters the place in a list with a reason.

Prerequisite: None. The code example comes from a React application tested with Vitest, and anyone who does not read TypeScript can follow the article without it. The principle applies wherever a piece of logic is moved into a function of its own and tested separately from its caller. As in the overview, test means an automated check that ends green or red, and test case means a written sequence of steps with expectations that a human runs at the screen.

Contents

The Bug: A Column Nobody Selected

In the DI² application, validation rules can be created for the columns of a table, for example a list of allowed values or a value range. This happens in a dialog. Its column field also offers columns that are not yet selected in the project and marks them with the badge “NEW”. If the user picks such a column, the application is supposed to select it in the project at that very moment. That is a write to the server, before the rule is even saved.

The screenshot shows the dialog in the test environment on 14 September 2026, that is, after the fix. The field is expanded, because only there is the badge visible, and with the click on a column the badge disappears again.

Dialog “Create new validation rule” with the column list expanded: CityID without a badge, the six other columns with the badge NEW.

No test and no agent found the bug. On 28 August 2026 the maintainer ran a manual test case and, in doing so, tried to move a validation rule to a different column. For that he picked the column StateProvinceID, which carried the badge “NEW” in the list. The dialog reported no error, the save button was active, and still the server rejected the rule because it referred to a column that was not selected.

The bug report from the same evening still left the cause open, and the next day it was confirmed. The reason lies in how the dialog recognized whether this call to the server is needed. Every column in the list carries an internal value. Columns that the application has captured once before carry their ID, a number. Columns that have never been captured carry their name instead. The dialog took this value as a hint: If there is a name, the column is new and has to be selected. If there is an ID, it is already selected and there is nothing to do.

That is not always true. A column that was selected once and then deselected again keeps its ID and is still no longer selected. The list showed “NEW” for it, because the badge asks for the actual state. The dialog saw only the ID, took the column to be selected and skipped the call. The server then checked the actual state and rejected the rule. On the server everything was in order, because the procedure behind the call explicitly re-selects a deselected column. The only thing missing was the call.

The Fix: A Function of Its Own and Twelve Tests

The fix came on 29 August 2026 at 13:37. Since then the dialog no longer checks what the value looks like but whether the column is selected. That is exactly what the badge “NEW” depends on too, so what the list shows and what the dialog does can no longer contradict each other. At the same time the fix moved this check out of the dialog into a pure function. That is the name for a function whose result depends only on its inputs and that changes nothing apart from that result. The closest thing in Postgres is a function declared IMMUTABLE. With that keyword its author promises that it neither reads nor changes a table and always returns the same result for the same inputs. A pure function therefore needs no dialog, no browser and no server, and a test can check it with prepared values in milliseconds.

The function receives the value from the list and all columns of the table and answers with a plan. The plan tells the dialog whether it can use the ID directly or first has to select the column on the server. Twelve unit tests call the function with prepared columns and compare the plan. In the example the function is called decidePick, and the test for the case from the bug report looks like this:

  1: it("a captured but deselected column must be selected", () => {
  2:   const columns = [
  3:     { id: "100", name: "CityID", isSelected: true },
  4:     { id: "101", name: "StateProvinceID", isSelected: false },
  5:   ]
  6:
  7:   expect(decidePick("101", columns)).toEqual({
  8:     kind: "select",
  9:     columnName: "StateProvinceID",
 10:   })
 11: })

Lines 2 to 5 contain the column StateProvinceID with an ID and as not selected, just as in the bug report. Lines 7 to 10 expect the plan select, meaning the selection on the server. There is nothing wrong with this test, because it sets up exactly the situation in which the bug occurred and checks the right result. The commit message did note, however, that the QA pass was still pending.

Moving the check out was the right thing to do. It does have a side effect that is easy to overlook: What gets tested now is the function, and the dialog that calls it is left without a test. Nowhere in the test above does the dialog appear.

The Question a Green Run Does Not Answer

37 minutes after the fix the report of the QA pass was in. It comprises eleven checkpoints, and all eleven passed. Most of them concern the server and neighboring cases. Two concern the call site: Does the dialog really issue the selection for the reported case, and does the same check fail against the state before the fix?

For the first question the QA agent wrote a component test. That is the name for a test that renders a piece of the user interface in a simulated browser environment and operates it like a user. The test rendered the dialog, clicked the deselected column and checked whether the call for the selection is sent, and against the fix the test was green. For the second question the same test ran against the state before the fix. There it failed, and the report records zero calls. Only this red run shows that the test actually detects the reported bug. The overview describes the method behind this as the counter-check against the old state, and the QA report calls it a negative control.

The two runs led to a further finding. It concerns regression protection, meaning the tests that are supposed to speak up when a bug that was once removed comes back. The report puts it roughly like this:

Gap in the regression protection: The twelve new tests check the pure function only. Anyone who turns the dialog back to the old check still gets the pure function green. Exactly the place where the bug sat is unguarded.

If someone puts the old check back into the dialog, say while tidying up or because an agent in a later session finds the earlier code simpler, the bug is back and none of the twelve tests reports it. The function still answers correctly, only nobody asks it anymore.

It has to be said that in this case no bug survived. The fix was right, and the fact that the dialog stood there without a test was noticed before anything broke there again. A human found the bug with a manual test case at the screen, and the question about the old state found that no test would notice it in future.

The QA agent did not add the component test itself to the project. The test served the report only, and what remained there was a recipe from which a permanent test can be written.

The Component Test at the Call Site

Eight minutes after the report and 45 minutes after the fix, such a component test was in the project for good. It came in a commit that is primarily directed against two related bugs in the same dialog. With that commit also came the first test file that renders the dialog at all. Until then not a single test had done so.

In total there are three tests at the call site. Each renders the dialog, clicks a column and looks whether the call to the server follows or stays away. Against the old check in the dialog one of them turns red, while the twelve tests of the function stay green. The other two cover cases that already worked with the old check, so that a later change does not break them unnoticed.

Such a test costs more than a test of the pure function, because it needs a stand-in for everything the simulated environment lacks, from the server down to individual browser capabilities. Such stand-ins are called mocks or stubs. That a stand-in of this kind can itself lock in a wrong assumption is the subject of the first route in the overview.

The QA report had already reported the bug as ready to close although proof was still missing. The QA agent had only written down the path at the screen as a manual test case and had not clicked through it itself, because the login was missing in its local environment. That test case had never run until then, and only in the afternoon did the maintainer run it. What the first run found in the test case itself is covered in the article The First Run Tests the Test Case, and why a written test case is not yet proof is covered in the article How to Write Manual Test Cases. By early evening the bug was closed.

Three Questions You Can Ask Without Reading the Code

A rule can be derived from the case: Anyone who moves logic out of a component such as the dialog into a function makes sure that the place where the bug sat stays protected too. That is the place that calls the function and acts on its result. Often a test at that place does the job, but not always, as the next section shows. Whether the place is protected can be settled with three questions, and none of them requires being able to read the test:

  • Where did the bug sit, and does that place appear in any of the new tests? In the dialog case the answer was no. A coding agent can give it in one sentence if asked.
  • Does a test turn red when that place is turned back to the old state? That is the counter-check against the old state from the overview. Its result is two runs, and both can be judged without knowing the language.
  • And if no test checks that place, is that intent or oversight? A green run does not answer that, because it looks the same in both cases. It can only be answered if the intent is written down somewhere, and the next section shows how the DI² project handles that.

What Became of It in the Project

In the DI² project it did not stop at the three tests for the dialog. The project gives its coding agents rules, and the rules for tests include a table that records for every test file what it secures. The row for the dialog’s new test file has since stated that three of its tests secure the wiring of the extracted function. By wiring the project means the connection between a building block and the place that uses it. Elsewhere this is called integration, and here it is about its smallest case, a function and its caller.

A word of its own such as wiring helps especially when working with coding agents. An agent starts every session without any memory of the previous one, but it reads what is in the rules. If it finds the word there with an example, it can recognize an unchecked wiring in the next task.

In the project that has happened six times since. Between 8 and 13 September 2026 six test suites were created that explicitly refer to this case. Five do so in the table, the sixth in a comment in its test file, and the count was taken on 18 September 2026. The pattern was the same every time: A building block was tested on its own, and nothing checked whether the place that uses it does so correctly. In one of these cases one building block supplied the address of a page and a second one displayed that address as a link. Both were tested, but no test checked whether the four views that are supposed to show the link actually pass the address on. Had one of them forgotten to, the link would have vanished there without any test turning red. The six suites do not stand for six bugs that slipped through but for the fact that the same gap keeps appearing and is now recognized in the project.

Not Every Connection Gets a Component Test

The answer to an unchecked connection is not always a component test. The project knows three further answers. For the link from the example, a test reads the source code of the four views and checks whether each of them passes the address on. A component test would be out of all proportion to the question here, because these views are between 1800 and 6100 lines long and a test would have to render them together with everything attached to them. Another connection is secured by the type check, an automatic check of the source code. If one of the functions that load the data for a view no longer returns a required field, the type check fails before the first run. In both cases a counter-check has shown that the safeguard actually triggers.

What Deliberately Stays Without a Test

The third answer is to leave a place without a test for now and to write exactly that down. The reason is in the same rules: Files that access the database do not get unit tests in the project for the time being, because the test environment for them is still to be built. This rule has a side effect. If a test is missing for such a file, nobody can tell anymore whether it is missing because of the rule or because nobody thought of it, since a green run looks the same in both cases.

Since 6 September 2026 the project has therefore kept a list. Anyone who deliberately leaves a place without a test enters it there and records why the test is missing, what a later change would destroy unnoticed and what is meant to close the gap one day. For everything that is not on the list the reverse applies: The test is missing by oversight. The first entry comes from the part of the application that writes to the Postgres database. There several records go into the database in one transaction, and each is protected by a SAVEPOINT of its own, to which the application rolls back with ROLLBACK TO SAVEPOINT if an error occurs. If someone removes these statements, a single faulty record aborts the transaction and Postgres discards all the records already written as well. In a counter-check on 6 September the statements were removed as a trial. By then the suite had grown to 1101 tests, and all 1101, the type check, the linter and the SQL check script stayed green. A further entry concerns exactly the pattern of this article, because there a function is tested and its call during saving is not. The list is thus the project’s answer to the third of the three questions.

The Same Holds in SQL: Function Tested, Procedure Not

What holds for the dialog holds in the database as well. A SQL function that checks, for example, whether a text is empty can be tested directly with a few SELECT calls. Whether the procedure that loads data calls this function at the right place or uses an older condition of its own instead only shows when the procedure runs against test data. The tests of the function stay green in both cases.

The DI² project knows the separation between checking a single building block and checking the interplay from its database development too. A check script reads every SQL file on its own and finds violations of the conventions. Whether all files together can be applied to an empty database only shows in a separate run against a throwaway database. That run has a limit too: Postgres does not resolve the SQL statements in the body of a PL/pgSQL procedure against the database when the procedure is created. Whether the tables and columns they address exist therefore only becomes apparent on the first call. The check script and the run are described in the article Why sqlfluff Can’t Lint Our SQL Conventions.

Working Prompts as Examples

The sessions in which the case was handled are not versioned. The two prompts here are derived from the bug report, the QA report and the commit messages and are not quotations. The general prompt for the counter-check is shown in the overview.

The first prompt hands a QA pass the question of whether any of the new tests would notice the bug:

The fix is in, and the twelve new tests are green. Check whether any of
them would report the bug if it came back: temporarily restore the old
check in the dialog and run the suite.
Acceptance: I see two runs, one with the old check and one with the fix.
If everything stays green in the first, a test is missing at the place
where the bug sat. Then write down which test that would be.

In the project the QA agent answered this question with the component test from the section above. It ran the test against the state before the fix, recorded the result with zero calls in the report and added the recipe for the permanent test.

The second prompt hands this recipe over to the implementation:

The QA report names a gap in the regression protection and a recipe for
it. Implement the three cases as permanent tests, exactly following the
recipe in the report.
Acceptance: Against the old check in the dialog the first case turns
red, and the twelve tests of the function stay green. Both runs are in
the commit.

Both prompts state what is to be checked and by what the result is accepted. They contain no file names, no libraries and no test technique, because the maintainer could not have specified those. Which building blocks have to be replaced in the test and how the dialog has to be opened was in the QA agent’s report.

Summary

  • Anyone who moves logic into a pure function afterwards tests the function and not the caller: The function gets fast and thorough tests, while the dialog that calls it stays without a test as long as nobody puts one there. That is how the situation behind the short form unit test passes, integration fails comes about.
  • Whether a test would notice the bug only shows in the run against the old state: Against the old check in the dialog one of three tests at the call site turned red and none of the twelve tests of the function. That result can be judged without being able to read the code, and in this case it was there before anything broke in the dialog again.
  • Since then the project specifically checks whether a building block is wired in correctly: Six later test suites refer to the case. Where a component test would be too expensive, a test reads the source code instead, or the type check secures the connection. What deliberately stays open is on a list with a reason.

FAQ

Why does a unit test not notice that the tested function is never called?

Because a unit test calls the function itself. Whether the place in the program that is supposed to use this function actually does so lies outside its field of view. If the bug sits there, the test stays green, and it stays green even when the bug returns after a later change.

Is high test coverage not enough?

No. Test coverage, or code coverage to be precise, measures which lines or branches were executed during the tests, not whether their result was checked. A coverage report broken down by file can show that no test touches a component, and that is a useful hint. But as soon as a test renders and operates the component, the lines it runs through count as covered, even if no test checks whether the decisive call takes place.

How do you test whether a function is actually called?

With a test that does not call the function but operates the place that is supposed to use it. A component test renders a piece of the user interface in a simulated browser environment for this, operates it like a user and observes whether the expected call follows. The server is replaced by a mock that records every call. Such a test is slower than a unit test and needs a stand-in for everything that is missing in the test environment.

Does every call site need a test of its own?

No. A component test pays off where a bug has sat before or where an extracted function decides whether a call to the server takes place. For very large components a test that reads the source code of the callers can be enough, and some connections are already secured by the type check. What deliberately stays without a test should be written down with a reason so that it can be told apart from a forgotten place.

Parent articles:

Sibling:

Control and conventions:

  • The Control Loop — how the QA pass and the counter-check against the old state became part of a recurring control when you cannot read the code.
  • Why sqlfluff Can’t Lint Our SQL Conventions — the same separation in the database: a check script for the single file and a run against a throwaway database for the interplay.