The Control Loop — How I Get AI-Generated Code Checked That I Can’t Read Myself

Reading code in order to control it takes two things: the language and the time. In the DI² project, both are missing. The maintainer comes from database development and knows neither TypeScript nor the web development the application is built in. And even someone who knows the language no longer reads through the amount of code a coding agent writes in a single afternoon.

That is not an admission of failure but a change in how the work is done. Anyone who has to verify AI-generated code without being able to read it needs checks that produce a result which can be judged without knowing the language. The agent can run those checks itself. Judging their result is up to the human.

This article describes what that loop looks like in the DI² project, what evidence there is that it actually runs, what it has found and where it has failed. Its most important find was a throttle against password guessing that was configured correctly and still protected no one. The skills, tests and audits are agent artifacts. The maintainer set the triggers, formulated the criteria and signed off.

The essentials up front:

• Control is a loop, not a review: Five stations run recurrently — sharpening rules, sharpening skills, retrofitting tests, declaring test cases, running updates and audits. Each has a trigger, an acceptance criterion and a place where its result lands.
• What can be checked is what has a result: A test run, a status code, a counting pass and an audit report can be judged without knowing the language. A line of code cannot. Whether a check checks the right thing is not settled by that.
• The loop came about after the fact: The project started in March 2026, the test foundation arrived in early July, the catalog of manual test cases at the end of August. None of this was planned from the start.
• The numbers prove that checking happens — not that it works: 28 rule files, 276 commits on them, 1,069 automated tests, 31 test cases. None of these numbers says the application is good.
• The loop applies to itself, too: When the bug backlog grew faster than it was worked off, suspicion fell on the new model generation. The measurement found a real break but could not attribute it: the share of bugs coming out of QA passes doubled, yet the model switch and the retrofitted test foundation fall on the same day. So what changed was what could be changed — two rules that appealed to judgment gave way to a limit that can be counted.

Prerequisite: None. No prior knowledge of Claude Code is needed, and terms are explained on first use. The article describes a way of working from a solo project and is not a recommendation for a team with its own quality assurance.

Contents

• How to Verify AI-Generated Code When You Can’t Read the Language
• The Loop: Five Stations
• What the Numbers Prove and What They Don’t
• What the Loop Has Found
• What the Loop Does Not Find
• The Loop, Applied to Itself
• What It Costs
• What Holds in the End
• FAQ
• Related Articles

How to Verify AI-Generated Code When You Can’t Read the Language

The usual answer to the question of how to control AI-generated code is: read and understand it before you take it on. That is good advice as long as its preconditions hold. It assumes that the reviewer knows the language, knows the libraries and that the amount stays manageable.

In the DI² project, none of these preconditions holds. The application is an ETL generator on Next.js and PostgreSQL. The maintainer can judge the database side, because that is where his professional experience lies. He cannot judge the frontend side, and in this project that is not going to change. A look at a React component tells him whether it looks tidy. It does not tell him whether it is right.

An uncomfortable conclusion follows. Anyone who cannot judge the language gains no control through sampling either. A sample is a control only if the reviewer could notice the error when there is one. Otherwise it is reassurance. For the part one cannot read, the question is therefore not how much of it to read, but which checks produce a result that is unambiguous even without knowing the language.

Such checks exist, and they share one property. They end with a statement that is true or false rather than with an impression. A test run is green or red. An endpoint answers with the expected status or it does not. A counting pass finds zero hits or more. An audit report lists findings with a severity. None of this requires the reviewer to understand the line that makes the difference. Unambiguous, however, means neither complete nor meaningful. A test run can skip tests, a report can carry a caveat, and both are in the output rather than in the verdict. A test can also check the wrong property and still be green, and that is written nowhere. Whether a check checks the right thing shows only in whether it comes out differently when there is an error. What happens when only the verdict gets read is shown further down by a case of its own.

The Loop: Five Stations

The five stations did not come into being at the same time, and they do not run at the same pace either. What makes them a loop is that each of them has a defined trigger and puts its result in a fixed place where it turns up again on the next pass.

First Station: Feedback Becomes a Rule

When the maintainer makes a correction, the work is not supposed to end with the correction. It is supposed to end with the correction being written down as a rule, so that the same intervention is not needed a second time. That does not apply to every correction, though, only to those whose repetition a rule would have prevented. A one-off factual error, a matter of taste or a case without a recognizable pattern gets corrected, and that is all. A rule made from it would lengthen the rule files without ever taking effect again.

The rules live as files in the project and are handed to the agent on every run. How these files are built and why there are many small ones instead of one large one is described in an article of its own. At the time of that article there were 27 files. Today there are 28, spread over 276 commits.

The trigger is a correction that a rule could have prevented. The rule is accepted when it names the concrete case that triggered it rather than merely stating a principle. The result lands in the rule file of the affected area.

At this point a qualification belongs that takes the station down a notch. That such a correction becomes a rule is an intention, not a mechanism. It is written in no rule file, no skill and no closing step, and nothing asks for it. In practice the maintainer gives the push, or the correction is so obviously recurring that it ends up in a file on its own. The 276 commits prove that it happens often, not that it happens reliably. The station therefore has exactly the weakness described further down for two bug rules: it appeals to judgment, and nobody counts.

What that looks like concretely is shown by a case from 29 August 2026. On two consecutive days, a debugging session had cost about an hour each in the wrong place, because two container worlds were running side by side on the development machine and the application hit sometimes one database and sometimes the other. What came out of that incident was not a sticky note but a mandatory step in two manual test cases: before the test run, check the schema version of the database. The test case text justifies this with the date of the incident and points to the note in which it was worked up. Since then, this step does two things. It prevents a failure on an outdated schema from being read as an error in the program. And in the second of the two test cases it prevents real data loss, because that case checks a protection mechanism which an outdated schema does not even contain.

That the lesson did not land in a rule file here is no oversight. The maintainer runs these two test cases himself at the screen, and a rule file is read only by the agent. The place follows from who has to follow the instruction. The same station knows the other case, too: after a subagent started 220 unthrottled compute loads for a verification on 4 September and made the workstation unusable for twelve minutes, that became a section in the rule file for tests. It names the incident with its date as the source and sets out four conditions for future load tests.

Second Station: Recurring Checks Become Skills

In Claude Code, a skill is a versioned file containing an assignment that can be invoked with a short command. For control, the versioned part is what matters. A checking assignment that is formulated afresh each time turns out differently each time. A checking assignment as a file is the same for months, and when it changes, the change is in the history.

The project has 14 such skills, and the two for control have been in operation since 16 April 2026. One runs a project-wide security audit, that is, a review of the whole project along a fixed list of areas to check. The other checks for updates to dependencies and to the components that run on the server. Stripped of the project-specific areas, the head of the security skill looks like this:

Role: experienced security engineer and red-team tester.

Run a project-wide security audit, independent of individual
features. Areas: dependencies, authentication and sessions,
injection, authorization, headers, secrets, rate limiting.

Goal: identify all security-relevant weaknesses and document
them in priority order.

Trigger: before go-live and after major changes.

Role: experienced security engineer and red-team tester.

Run a project-wide security audit, independent of individual
features. Areas: dependencies, authentication and sessions,
injection, authorization, headers, secrets, rate limiting.

Goal: identify all security-relevant weaknesses and document
them in priority order.

Trigger: before go-live and after major changes.

Three properties of this matter more than they look. The assignment names the role, so that the agent does not stay in the mode of the explainer but works in that of the reviewer. It names the trigger instead of asserting a calendar rhythm. And it demands prioritized documentation, that is, a result with severities instead of an assessment in running text.

The second checking skill carries a property that matters even more for control. Its description states in so many words that it changes nothing by itself. It checks, writes its findings into a dated report and keeps a history of which updates were applied and which were deferred. What happens as a result is the maintainer’s decision. A checking tool that may change what it checks will, on its next run, check its own correction, and then there is no longer any way to say which state it checked.

The trigger of this station is a check that is needed for the second time. The skill is accepted when its assignment names the role, the trigger and the form of the result. The place is the skill file in the project. What the skill produces on each run belongs to the fifth station.

The weakness of this station is the aging of the assignment. The security skill has been unchanged since 25 April, and its section on login names the application’s library, not the identity provider that has been in front of it since 18 April. That the August audit found the dead throttle anyway is because the list of areas is general enough, and not because anyone had brought the assignment up to date.

Third Station: Retrofitting Tests Under Existing Code

The project started on 19 March 2026. The test foundation came into being on 2 July, just under three and a half months later. It runs on Vitest, a widely used test tool for JavaScript and TypeScript that runs the tests in the terminal and reports green or red for each individual check. From the start, the foundation separates two kinds: unit tests for pure functions and component tests for the React interface, for which Vitest rebuilds the browser environment in memory.

Specification, technical design and implementation fall on the same day and brought 6 test files with 57 tests. The broad coverage followed the next day in three steps and ended at 28 files with 566 tests. Before that there were no automated tests. The loop was not a plan, it is a reaction.

Retrofitting tests under existing code has a trap that cannot be seen without knowing the language. A test written against the current state is green as soon as it describes the current behavior. Whether that behavior is right, it does not say. It only cements it. For the tests of the foundation this largely holds: they were written against the code that ran in July, and they only trip when its behavior changes, regardless of whether the change is right or wrong. An acceptance criterion that does more exists for the case that has been the most frequent since then, the test for a fixed bug. It can be checked even by someone who cannot read the test:

Write a test that covers this bug.

Acceptance criterion: The new test must be RED on the old state,
i.e. before the fix. If it is green there, it does not check what
it claims to check. Show me both runs.

Write a test that covers this bug.

Acceptance criterion: The new test must be RED on the old state,
i.e. before the fix. If it is green there, it does not check what
it claims to check. Show me both runs.

(This prompt is reconstructed from the assignment and its result, not logged verbatim. The skill excerpt above is in the repository in this form, translated here from German.)

The demand to show both runs is the actual core. It turns a claim by the agent into two outputs the maintainer can see side by side. Red on the old state, green on the new one, and between them lies exactly the fix.

The criterion presupposes that the bug can be reproduced. With a bug that occurs only occasionally, the old state shows red sometimes and green sometimes, and then the comparison does not hold. And it only proves that the test sees this one bug, not that it fully covers the requirement behind it. Since the foundation, the trigger of this station is a bug that gets fixed. The test is accepted with the two runs, and afterwards it lives next to the code it checks.

Fourth Station: Declaring Manual Test Cases

Not everything can be checked automatically. In this application, the user can approve a source table, that is, declare its modeling complete. After that, nobody is supposed to change anything about it anymore, not even the validation rules of its columns, and not even the owner of the project. Whether that lock takes effect on every path the interface offers is, in the end, checked by a human.

So that this check, too, produces a result rather than an impression, the cases are written down beforehand: step by step, each step with its expectation next to it. The catalog came into being at the end of August 2026 with 22 cases and stands at 31 today.

The trigger of this station is a behavior that no automated test reaches. A test case is accepted when every step has its expectation next to it. The place is the catalog in the project, and every run records its result in a log.

The benefit shows even before running anything. Whoever writes down a test case has to formulate the expectation, and in doing so it becomes apparent when there is no unambiguous expectation at all. Besides the steps, every case carries a goal, a precondition and a cleanup step, and its execution is recorded separately from it, one log per run.

A test case needs the same upkeep as a rule. On 30 August one was adjusted after the expected behavior had changed. The commit message names the reason: avoiding a false failure. A test case that stays unchanged after a behavior change reports errors from then on that are not errors, and becomes worthless as a control.

Fifth Station: Running Updates and Audits

The last station is the only one driven from outside. Dependencies get security advisories, the server’s operating system gets updates, and both happen regardless of whether anyone is working on the project right now. Five dated security reports sit in the project, plus the audit file that records every run with its date, the areas checked and the findings. The trigger comes from outside, as a security advisory, or from the skill, which names delivery and major changes. A run is accepted when its report lists the findings by severity and the maintainer has decided what of it gets implemented. The place is the reports and the audit file. The weakness of this station appears further down as a case of its own: a check report can carry a verdict whose caveat gets lost in the handover.

The five stations at a glance, with the weakness each of them has:

What the Numbers Prove and What They Don’t

The numbers from the project are quickly listed, and that is exactly why a warning is needed. As of 10 September 2026: 14 skills, 28 rule files with 276 commits, 85 test files with 1,069 tests, 31 manual test cases, 5 security reports.

None of these numbers says the application is good. They say the loop is running. That is a difference easily overlooked, because large numbers look like quality. A suite of 1,069 tests can miss the one property that matters. A catalog of 31 test cases can lack the one case that goes wrong.

More interesting than the final counts are therefore the movements. The automated tests stood at 566 in early July and stand at 1,069 now. The test cases have grown from 22 to 31 since the end of August, nine in just under two weeks. The rule files have grown by a single one since the article that describes them, their commit count by considerably more. This movement is the most telling of all. It means that hardly any new topics are added, but the existing rules are being sharpened continuously.

Nor is there a fixed interval for the audits. The five security reports lie roughly a month apart, which matches the figure given in the infrastructure article. The audit file itself, by contrast, records six runs in eight days between 30 August and 6 September. Both together give the honest answer: the pace is set not by the calendar but by how much has moved.

What the Loop Has Found

The most telling find comes from a security audit on 28 August 2026. For login, the configuration of the upstream server contained a limit on requests per minute. It was written correctly, it was active, and anyone reading it concluded that the login form was protected against automated password guessing.

The limit sat on the endpoint of a login procedure the application used to handle itself. After the switch to a dedicated identity provider, that endpoint no longer existed. The throttled path was never called, and so there was no effective throttling anywhere. The finding in the audit records why that is worse than no protection at all: an existing throttle reassures every check that looks at the configuration.

What matters for this article is which kind of check delivered the find. Reading code would not have delivered it, because the file was, after all, correct. What was needed was a comparison of two things that lie in different corners of the project: the throttled path in the server configuration and the list of login procedures actually set up in the application. Exactly such comparisons across area boundaries are the strength of a project-wide audit, provided its assignment names both areas. The skill head above does, since authentication and rate limiting are on the same list there. The assignment alone, though, does not prove that the comparison took place. Only the report proves that, when it lists both locations side by side, and the finding of 28 August does: the throttled path from the server configuration and the list of configured login procedures from the application. The comparison itself does not depend on whether the person commissioning it can read the language.

What the same incident looks like from the diagnostic side, and how its effect was subsequently measured rather than believed, is described in the article on debugging with a coding agent.

What the Loop Does Not Find

The loop has blind spots, and three of them are documented.

It does not find code that deviates from the rules. The rules are in files the agent receives on every run, and still it does not always stick to them. In this project that became visible in 799 hardcoded font sizes. That was no legacy from before the rule: the font-size scale had been in the rule file since April, and nine out of ten of these spots came into being only afterwards, while the agent did use the scale in other places. The consequence was not to phrase the rule better. It was to make the rule machine-checkable. How that works when no ready-made tool covers the convention is shown by the custom guard for the SQL conventions.

It does not find a success message with nothing behind it. A rollout run printed two screens full of checkmarks. Afterwards, the configuration was active nowhere. The copy command had landed in the wrong directory and done nothing, and the setup script sets its checkmark without checking whether the command before it had succeeded at all. It was noticed only because a line was missing from the output that should have been there. The agent has its share in this. It had handed over the rollout commands without pushing the fix to the central repository first. So the server fetched the old state during the rollout, and the commands could not take effect at all.

It does not find the caveat that sits in its own check report. On 29 August, a QA pass delivered a careful result across eleven check items, with a reproduction of the bug and a counter-check against the state before. The verdict read “passed, ready to close”. But the same report also said that part of the check could not be run locally and was therefore only written down as a new test case. It was never run. The next agent adopted the verdict and carried on without looking whether that test case ever shows up in a run log. It was noticed through a question from the maintainer.

The last point applies more generally. When a check result is passed from one step to the next, the judgment arrives and the caveat does not. A caveat that sits in the middle of a report’s running text does not survive its summary. And whoever reads the summary takes it for a fact, because it comes from someone who looked more closely than they did. Humans make the same mistake with check reports. What is new is how fast and how casually it happens here. The countermeasure costs almost nothing: a verdict names not only a status but carries the open items along as a condition, in the form “passed, provided X still runs”.

From this follows the most uncomfortable sentence of this article. The control exercised by the agent needs a control of its own, and that cannot again consist of asking the agent.

The Loop, Applied to Itself

In early September it became apparent that the loop was generating work instead of reducing it. On a single day, 21 new bug entries were added and none was closed, with 26 open, among which there was not a single case of the levels critical or high. A bug backlog that grows without anything serious in it is not a quality signal. It is a process problem.

The obvious suspect was the model. In early July, a new model generation came into use with Claude Fable 5, and it seemed to check considerably more precisely and therefore find more. Instead of debating that suspicion, it was measured. The result cut both ways.

The number of new entries per commit stayed practically the same across all models used. What changed is which kind of entry comes into being. Until the end of June, just under a third of new bugs came out of a QA pass, from July almost two thirds, in August three quarters. The share of entries that explicitly name a predecessor doubled from 9 to 18 percent in the same period. It is not down to Fable alone: Opus 5, which ran alongside it from August, shows the same pattern. And because these entries come from QA passes, the increase first says something about the checking and not about the writing: more actually gets found. Whoever looks only at the total sees none of this — the first measurement had exactly that flaw.

The break is thus documented, its cause is not. On the day the new model generation was first used, the test foundation was retrofitted as well. Since then, checking has been systematic, and what is checked systematically yields findings, no matter who wrote the code. Both explanations fit the same numbers, and no measurement on this dataset can separate them. The honest answer is therefore not that the model is cleared, but that the question stays open.

One share could be named independently of that, and it lies in the process. There was already a rule limiting what gets fixed. There was none limiting what gets recorded at all. Every observation while retesting a fix became an entry of its own, each of those could spawn a successor in turn, and from one root entry ten descendants grew this way over six generations. Measured on the day of the analysis, 72 of 594 entries came from such a chain, and two roots had produced 18 descendants between them.

The consequence is written verbatim in the new rule: the two existing rules appealed to judgment and were overridden anyway, and chain depth is the only threshold that can be checked. Since 6 September, a follow-up entry may no longer spawn a successor of its own. The chain ends after two generations, the root and one rework. The predecessor line sits in the header of every entry, and that is why a script can check this limit.

A successor, here, is an entry that arises while retesting a fix and carries that fix’s number as its predecessor. The limit suppresses no bug, it demands a fresh justification. A finding beyond the second generation is recorded as a standalone entry with its own description if it is severe enough, and otherwise noted as a line on the predecessor. Whoever cannot explain a finding without referring to its predecessor has not found a new bug but is continuing the chain. That it is two generations and not three is a setting for this project, not a rule claiming general validity. What transfers is the form of the limit, not its value: a threshold a script can check, instead of a judgment call.

It is the same lesson as with the drift finding, only this time on the project’s own process instead of on the generated code. A requirement that demands judgment gets overridden under time pressure, by the agent as well as by the human. A requirement that can be counted holds, because a script can enforce it and nobody overrides it unnoticed. Whether it is the right requirement, that does not decide. Whoever sets up a control loop should therefore ask, at every station, whether its criterion can be checked by counting or merely sounds plausible. And then the second question, whether the criterion comes out differently at all when there is an error.

What It Costs

The price of the loop is not building it but maintaining it. Every rule has to be kept current, otherwise it checks against a state that no longer exists. Every test case has to be carried along after a behavior change, otherwise it produces false failures. Every check report wants to be read, and the 276 commits on the rule files are exactly this work in numbers.

Then there is the slowdown. A check that has to run before every delivery and stops it on red costs time, and its benefit is invisible as long as nothing happens. The temptation to skip the check just once is greatest on the day it would be needed most.

The real point, though, is a different one, and it connects this article to the overarching thesis of this branch: the work does not disappear, it shifts. In a team with its own quality assurance, part of these tasks would sit with project management and testers. In a solo project, all of it lands on the same person, and on top of the work the agent has seemingly taken off their hands. Whoever books the saved writing time as a gain without setting the control time against it is getting the sum wrong.

What Holds in the End

• What can be checked is what has an unambiguous result. Whoever does not read the language needs controls that trip on their own when there is an error, instead of waiting for an attentive eye.
• A criterion that can be recomputed holds. One that merely sounds plausible does not. Holds means: it gets complied with, even under time pressure. That applies to the generated code as much as to the project’s own process, and it is documented in both places. Whether it is the right criterion, that does not say, and that is what the next point is about.
• Numbers prove that checking happens — not that it works. A growing test base says that work is being done. Whether the right thing is being checked is answered only by the question whether a check actually comes out differently when there is an error.
• A verdict without its caveats is incomplete. Whoever passes on or receives a check result should carry the open items along as a condition, otherwise the assessment travels on alone.
• The loop does not replace understanding. It replaces reading as a control. The difference matters because it marks the boundary: it catches what can be checked, and it catches nothing for which someone would have had to ask the right question.

What transfers from this is the form, not the values. A trigger, a checkable criterion and a place for the result can be set for every station in any project, as can the counter-check whether a check comes out differently when there is an error. The numbers, the two-generation limit and the cut of the skills, by contrast, are data from one project and are no use as a template for another.

FAQ

Related Articles

Parent articles:

• Agentic Coding from a User’s Perspective — the sub-hub of this branch and the thesis that the work shifts rather than disappears.
• AI-Assisted SQL Development with Claude Code — the entry point to rules, skills and agents.

The loop’s artifacts, one by one:

• 27 Rule Files Instead of One CLAUDE.md — how a rule file is built and why there are many small ones.
• Deriving SQL Conventions with Claude Code — how rules emerge from an existing codebase.
• 799 Hardcoded Font Sizes — the drift finding that shows why rules alone are not enough.
• Why sqlfluff Can’t Lint Our SQL Conventions — making a convention machine-checkable when no tool covers it.

Sibling articles:

• One VPS, Four Environments, No Cookie Banner — the infrastructure that is the subject of the fifth station’s audits.
• Two Agents, One Working Tree — what goes wrong in parallel operation, including the load-test incident from which this article’s test rule emerged.
• The Agent Measures Where I Click — Debugging Postgres, Docker and a Swapping Server with a Coding Agent — the same incidents from the diagnostic side: why a finding only holds with a measurement behind it.