The Agent Measures Where I Click — Debugging Postgres, Docker and a Swapping Server with a Coding Agent

Published Sep 11, 2026
by

A bug gets reported the way it feels: the message does not appear, the page barely responds, a tab no longer loads after a restart. A user can only describe what the interface shows, and the interface is the end of a long chain. A coding agent with access to the database, the container runtime and a shell does not see the interface and starts debugging somewhere else. What it sees instead is what has been stored, what has been started and what has been swapped out.

This article describes four debugging sessions from the private DI² project in which exactly this difference decided the outcome. In three of them, a measurement settled early what a guess would have left open: which part of the chain is affected, whether the new code can be to blame at all, and whether a protection actually takes effect. In one case, the same method spent an hour ruling out explanations without hitting the cause. The same agent then built an explanation from the remaining signs that was internally consistent and still wrong, and presented it as a finding with more confidence than the evidence allowed. Both sides belong in the same article because they show the same way of working, once at its strength and once at its limit.

All four cases arose during development, not while running an application with users. Development here does include the maintainer setting up the project’s four environments and keeping them running, and a database developer does not come with a server background for that. The agent proposed the commands for it, and they were only executed after the maintainer had approved them.

The essentials up front:

  • The first question is not “why” but “is the suspicion even right”: Refuting a suspicion is easier than confirming a cause. Confirmation requires tracing the whole chain. Refutation is often a single query, and an agent with access to the database and the server should start exactly there. In one of the cases, a comparison across four environments disposed of the suspicion that the newly deployed version was to blame within a few minutes, without anyone reading a line of code.
  • The database tells you whether something was never written or just not displayed: Many applications store messages in the database first and only display them from there. If a message is missing from the interface, it may have been lost while being written to the database or only on its way to the display. A look at the record and its timestamp settles that immediately. If the message is in the database, everything before storage is ruled out, and the search starts behind it.
  • Tools have blind spots you need to know: A service whose memory has been swapped out to disk almost entirely shows up in the container statistics with a small value, because they only report the share in RAM. Anyone who knows the service’s normal footprint may pause at that, but a small value easily reads like a healthy one. How much has actually been swapped out only shows up in the machine’s memory overview.
  • A step that ran is not a step that worked: A brake on automated password guessing had sat correctly in the configuration for months. It counted requests on the old login path, however, which no longer existed after the switch to an external login service, and the actual login route ran unthrottled. A different fix removed a symptom without hitting the cause, so that everything was back the next day. In both cases the only thing that helped was measuring the effect directly instead of trusting the configuration file or the success message.
  • The flip side belongs here too: A language model effortlessly tells an explanation that sounds internally consistent even when the evidence does not support it. That is exactly what happened in one of the four cases. The most useful follow-up question to an agent is therefore not “why” but which concrete measurement supports this statement.

Prerequisite: Basic database knowledge and a rough idea of what a container is are enough. The cases are told so that the diagnostic path stays easy to follow, not the application behind it. Commands appear as patterns, not as instructions to copy onto someone else’s system.

Contents

What the Agent Sees and the Browser Does Not

An error report from the browser is an observation at the end of the chain. Between the user’s action and what they see lie a form, a server action, a database transaction, a notification path and a rendering step. Any of these stations can be the cause, and the report does not say which one it was. Anyone who only has the interface has to guess at which station to start looking.

An agent working in the same environment has different instruments. It can ask the database directly whether a record exists and when it was written. It can check which containers are running and which of their ports are actually reachable from outside. It can open a shell on the machine and look at how much memory is really available. This access does not make it smarter than the human. It makes certain questions cheap that used to be expensive, and cheap questions get asked earlier.

The difference shows most clearly in refuting. Confirming a cause is laborious because you have to trace the whole chain. Falsifying a hypothesis is often a single query. That is the lever in the four cases that follow. In none of them did the agent guess the cause. In three, a measurement decided early where to keep looking, and in one, the same method spent an hour only ruling things out without hitting the cause.

Case 1: Is the Message Missing, or Just Not Displayed?

The maintainer reported that an expected error message did not appear in the interface. The application first stores such messages as a record in the database, and the interface displays them from there. The observation from the browser therefore leaves two entirely different causes open. Either the message was never written to the database, in which case the fault lies in the write path. Or the record is there and just does not reach the interface, in which case the fault lies on the way from the database to the display, in the notification path. Both guesses are equally plausible, and they lead to completely different places in the source code.

Instead of following the bug description, the agent looked in the database first. The message was there. It had been written 22 seconds after the triggering change, in two write operations 348 milliseconds apart. That settled the question before anyone had read a line of application code. The write had worked, the display had not, and the search moved to the notification path. That is where the cause was.

The pattern behind it is simpler than it sounds. If a bug is reported as “something is missing from the display” and the application stores the affected data before displaying it, there is a measurement that immediately decides whether to look before or behind the storage step. For that, the query has to reach the same database the application reads from. Case 2 shows that even this can go wrong. As a rule of thumb, the measurement looks like this:

Ask the database whether the expected record exists,
and read its timestamp.

It does not exist  -> the fault lies in the write path.
It exists          -> the fault lies between storage and display.

The timestamps are more than a by-product, even if on their own they do not explain anything. What the agent had measured was only that the record was written twice, 348 milliseconds apart, and that this happened 22 seconds after the triggering change. The timestamps do not say why. Two write operations can stem from a second write path or from a retry, and a delay from a background job that only runs at intervals. Gaps like these, though, are often the cheapest way to reconstruct a sequence nobody remembers.

Case 2: Two Hours Down the Wrong Rabbit Hole

After the development machine was restarted, one of the application’s tabs no longer loaded. The error messages alternated between “relation does not exist” and “column does not exist”, and on top of that a test project was missing that had still been there the day before. Everything about it looked like damage to the database or a bug in the application.

The agent formed hypotheses one after another and checked each one with a measurement. Among other things, it compared the instance identifier (system_identifier) across both access paths, a number PostgreSQL generates when a database instance is set up and that the instance keeps from then on. It was identical on both paths. The database log showed not a single error since startup, the activity view showed no connection from the application, and the agent read the connection address directly from the environment of the running process rather than from a configuration file. Each of these checks was quick, and apart from the identifier comparison each one ruled out an obvious explanation. Even so, after an hour no cause had been found.

It was the human who found it, and only in passing. The maintainer threw in two observations that no tool shows: the test project had still been there the day before, and the database normally ran on its default port. That turned attention to the container’s port forwarding. A container is sealed off from the outside, and for the application to reach its database, the machine forwards one of its own ports to the container. This forwarding was configured but not active. The port was held by two forwarding processes that looked like leftovers from an earlier run. They were terminated, the application worked again, and the case was considered solved.

The Fix Worked and Still Missed the Cause

The next day everything was back. The same symptoms, the same hour of debugging, and this time the investigation went deeper. Two complete container worlds were running side by side on the machine, meaning two Docker installations, each with its own engine, the service that actually runs the containers. One was a system engine, the relic of a first installation whose removal had remained incomplete, and it was set up to start with every boot. The other was Docker Desktop, the installation that was actually intended. Both brought up a stack with the same name, including a database with the same container name.

That creates a race for the port. Only one process can hold a given port on the machine, and whoever starts first gets it. In this setup, the container of the other world still started, just without forwarding, and its database was then unreachable from outside. The application connects through the machine’s port and ends up in whichever world holds it. After a boot, that was the system engine, because it came up first.

The cross-checks the agent used to inspect the data ran along a different path. They were queries via docker exec, which run directly inside the database container and bypass the port. Which engine a docker command talks to is normally determined by the active Docker context, a setting of the command-line tool, and the active context was Docker Desktop’s. There was a container with exactly the same name there, and the cross-checks ran against its database without anything indicating it. The system engine was not hidden. It was simply not addressed by any docker command, and that is why its containers did not show up in any list. So the two data sets slowly drifted apart, because clicks and scripts hit one database or the other depending on which world held the port.

That turns the supposed fix from the day before into something other than what it seemed. Terminating the orphaned processes did work, because the application ran again afterwards. It did not hit the cause, because the second installation started again with the next boot. The fix even concealed the actual problem, because it removed the symptom reliably enough to end the search. Only the relapse showed that the supposedly orphaned processes belonged to the system engine and were therefore a symptom, not the culprit. The case was only resolved once the system engine had been shut down and uninstalled.

Spotting Two Docker Engines Earlier: The Rules From the Second Incident

Two rules made it into the project documentation. The first concerns the order: the question of which engine the active Docker context addresses belongs at the start of every local diagnosis, before any listing of containers. docker context ls shows all configured contexts with their endpoint and marks the active one with an asterisk. If the DOCKER_HOST environment variable is set, it overrides the context, and the list then says so explicitly. Several contexts are only a hint, because a context can also point to a remote machine or to the same endpoint as another. If the asterisk is on the wrong context, though, everything measured afterwards is worthless.

The second rule concerns the proof of identity: when the interface and a cross-check show contradictory data, the instance identifier can be used to check whether both paths reach the same database instance at all. An instance carries exactly one identifier. If it differs between the two paths, the paths demonstrably reach two different instances, and every further cross-check along the wrong path checks the wrong database. The reverse does not hold. A physical copy of the data directory, for example from a backup with pg_basebackup, carries the same identifier as its original, so an identical identifier does not rule out two instances. If a logical dump is restored instead, the target instance keeps its own identifier.

Query the instance identifier over both paths and compare:

  Path 1: docker exec directly in the container (engine of the active context)
  Path 2: through the machine's port, the way the application connects

  SELECT system_identifier FROM pg_control_system();

Different values  -> two different instances, proven.
Same values       -> no proof, a physical copy carries the same identifier.

One detail cost a lot of time and is worth mentioning. As long as the other engine held the port, restarting the container did not bring the forwarding back either. Anyone who tries that takes the container for broken, while the problem lies with the engine that occupies the port.

Case 3: “The Page Is Barely Responding”

Immediately after three deployments in a row, the maintainer reported a vague symptom. The prompt was a single line:

the page is barely responding. can we measure anything

One suspicion was obvious: new code had just gone out to three environments, so presumably the newly deployed version had broken something that used to work. The agent did not chase that suspicion in the code but compared the environments. It first called a static version route across all four environments and then the start page and the login page in each environment. The result disposed of the suspicion at once, because all environments were affected, including the production environment, which was not running the newly deployed version at all. That ruled out the newly deployed version as the cause, and the search moved to what all four environments share.

The decisive step lay outside the application, in a shell on the machine itself. The system load, which uptime reports as a measure of how much the machine currently has to do, was low. That made it unlikely that processing power was the bottleneck. At the same time, 2.9 gigabytes sat in swap space on disk, although 5.6 gigabytes of RAM were free. The identity provider, the service that handles login, was hit particularly hard. Of its 802 megabytes, the system had swapped 754 out to disk, and only 48 were still in RAM.

Swapped-out memory only comes back when a process needs it again. Linux does not bring it back on its own just because space has become free. That is why it stayed on disk even though there had long been room again. That fits the symptom: the first login request took 9.64 seconds, every further one only 0.19 seconds. On first access, the system has to fetch the swapped-out parts back from disk, and after that they are in RAM again. An ordinary cold start produces the same pattern, though. What pointed to swapping as the cause here was that almost all of this service’s memory was demonstrably on disk.

A tool trap is an inseparable part of this case. The container statistics that docker stats displays only report the share in RAM, not the swapped-out share. The affected service appeared there with 48 megabytes. Anyone who knows that it normally uses around 800 megabytes could have paused at that value, because it is not inconspicuous but suspiciously low. With a sluggish service, however, you look for a value that is too high, and a small one then reads more like a healthy one. Even then, pausing only gets you a suspicion. The gap to the normal footprint can have other causes too, such as a service that has just been restarted. How much has actually been swapped out on the machine only shows up in its memory overview, for example with free -h, because the docker stats table has no column for it.

For sluggish services despite low load:

  docker stats   -> shows only the share in RAM
  free -h        -> also shows the machine's swap space

The telltale sign is all three together:
  first request takes seconds, second is instant
  low system load
  swap space in use while RAM is free

The agent did not carry out the fix itself. Bringing the swap space back is a system-wide intervention, and the maintainer’s private data lives on the same machine. The agent handed over the command sequence and put a pre-check in front of it that calculates whether the swapped-out memory fits into the free RAM at all. If it does not, the kernel mechanism that kills processes when memory runs out kicks in, and a sluggish service becomes a dead one. The request looked roughly like this:

Give me the command sequence to bring the swap space back.
Put a calculation in front of it that checks whether the swap in use
fits into the free RAM, and that aborts if the reserve is too small.
Also tell me what prevents this state permanently.

(The second prompt is reconstructed from the task and its result, not taken from a log. The first one appears above as it was typed, translated from German.)

The maintainer then ran the command sequence, and only its result confirmed the explanation. Login dropped from 9.64 to 0.19 seconds and swap usage to zero, without a single line of application code. To make a repeat at the next memory spike less likely, a kernel parameter was adjusted as well. It weighs whether the system, under memory pressure, prefers to swap out process memory or to drop the file cache, and it was lowered from its default of 60 to 10. That makes the kernel more inclined to sacrifice the file cache instead of pushing a service out to disk. It is a weighting, not a guarantee. If RAM runs short during a spike, the kernel swaps anyway, and whether the same value helps on another machine depends on what occupies the memory there.

One detail of this case is uncomfortable and still belongs in the story. The agent had long since produced the first hint itself without following it up. Its own first measurement call had taken 5.16 seconds, the second 0.10. That is exactly the pattern it later found again on the machine. At first it dismissed it as a cold start. That was possible, but it never checked that interpretation.

Case 4: The Rate Limit That Never Kicked In

The reverse proxy configuration contained a limit on requests per minute for the login path. Anyone reading it concluded that login was protected against automated password guessing. The line was there, it was written correctly, and it had been active for months.

The limit sat on the endpoint of a login procedure the application used to handle itself. Since the switch to an external identity provider, that endpoint no longer exists. The throttled path was therefore never called. There was no effective throttling of login anywhere, and the dangerous part is not the missing throttle. It is the existing one, because it reassures every check that looks at the configuration.

The fix put the limit on the identity provider’s login path, and this time its effect was checked independently of the configuration. The endpoint was actually hammered with requests, and after only a few attempts it answered with the status for too many requests. A cross-check against a neighboring endpoint that must not be throttled kept returning regular responses. That established that the throttle takes effect on the login path and does not slow down the neighboring endpoint.

In addition to the throttle in the proxy, there is a second line of defense in the identity provider itself. It temporarily locks an account when too many logins for that account fail in a row. Since the fix, the setting for this is in the project’s setup script. Whether it is active was nevertheless not inferred from that script but queried directly from the running identity provider for all four environments. A script says what is supposed to be set, not what actually is set.

The two lines complement each other because they hit different attacks. The throttle in the proxy counts per source address and slows down the simple case in which someone tries many passwords from one place. If the attempts come from many addresses, it is not enough on its own. The lockout in the identity provider counts per account instead, and therefore also takes effect when the attempts against the same account come from many addresses. Either one alone would have looked like a solution and left one of the two cases open.

On the same day, the same class of error occurred a second time, this time while rolling out the fix. The rollout run printed two screens full of checkmarks, and afterwards the configuration appeared to be active in all four environments. In fact, the copy command came to nothing because it ran in the wrong directory, and the setup script ran in its old version. The effect was zero. The script prints its checkmark regardless of which status the call before it returned. The checkmark testifies that the line after it was reached, and nothing else.

It was only noticed because one specific expected line was missing from the output. It was not the overall impression that revealed the error but a targeted search for a single sentence that should have been there. The agent caused this third instance itself. It had handed over the rollout commands without pushing the corresponding commit first. The commands could not work on the server at all, because the code there did not yet contain the fix.

A Step That Ran Is Not a Step That Worked

At first glance, Case 2 and Case 4 do not look related. One plays out on a development machine and is about containers, the other on a server and is about security configuration. They still share the same structure, and that structure is the real yield of the four cases.

In both cases there was a signal that testified to the step, and nobody checked the effect. The orphaned processes were terminated, so the problem was solved. The throttle was in the configuration, so login was protected. The rollout script printed its checkmarks, so the fix was active everywhere. Each of these signals was true in itself. The process was terminated, the line was in the file, the script had run to completion. Between signal and effect, the chain had still broken, and none of the signals noticed.

What helped in every case was the same recipe. You look for a second, independent route to the same question, ideally one that turns out differently when the effect is missing than when it has occurred. For a configuration, that means hammering the endpoint and reading the setting back from the running system instead of reading the file. For a fix, it means restoring the condition under which the error occurred, and after a startup problem that includes another restart. For a rollout, it means searching for one specific expected line instead of judging the overall impression.

This kind of cross-check is easy to automate, which makes it one of the places where an agent can do more than a visual check. It has to be asked to, though. On its own, it does not replace the success signal with an independent measurement, and an agent that is only asked whether the change has been rolled out reads the same checkmarks as the human.

The Flip Side: When the Agent Tells a Story

Case 2 holds a second lesson, and it is less flattering. In the middle of the fruitless search, the agent built an explanation from the available signs in which everything fit together, and presented it as a finding. The explanation was internally consistent, it contradicted none of the measurements, and it was wrong. Only when the maintainer asked what exactly it meant did the agent take it apart and separate which part had been measured and which part interpreted.

That was not a slip but lies in how a language model is built. It produces fluent, plausible text, and an explanation of a cause is exactly that: fluent text. Forming a fourth statement from three observations that feels like a conclusion costs a language model practically nothing. Backing up the same statement costs it just as much as anyone else. The whole difference lies between these two steps, and it cannot be seen in the finished text.

For the user, that means splitting every finding into two parts. The first part is what was measured, and it can be named: a query, a return value, a timestamp, a status code. The second part is what was inferred from it, and that is a claim. The most useful follow-up question to an agent is therefore rarely “why”. It is which concrete measurement supports this statement, meaning which query, which return value, which timestamp. If the answer points to a concrete measurement, the finding holds. If it points to a synthesis of several clues, the finding is a hypothesis and should be treated as one.

A second limit concerns not interpreting but acting. The swap space from Case 3 would have been technically within the agent’s reach. It still did not carry out the intervention, because the same machine holds data that does not belong to the project. Whether a system-wide intervention runs on someone else’s data is not a decision an agent makes alone, and that holds even when the proposed command is correct. The agent delivered the proposal along with the safety calculation, and the human executed it. That is exactly how the work should be divided.

A related side finding belongs here too, because it came up in the same week. When two sessions share a working tree, a failed test run is at first only a hypothesis and not yet a bug in the product, because it may just as well stem from the other session’s work. What follows from that for parallel operation is covered in Two Agents, One Working Tree. And that, conversely, a green test run is also only a signal and not proof is a topic of its own, which a later article on this blog about test-case discipline will take up.

What Remains

Together the four cases cost about four hours of debugging, and most of that went to the two incidents with the two Docker engines. Four things connect them.

  • The first measurement should narrow the search, not deliver the cause. Does the record exist? Are all environments affected? Do two access paths land in different database instances? Questions like these do not yet say what is broken, but they rule out whole areas. They cost almost nothing and therefore belong at the start of every search.
  • The most obvious suspicion gets a cross-check first and an investigation only afterwards. Whether the last change is to blame at all can be settled in minutes. Anyone who skips that question and goes straight to the code risks hours in the wrong place.
  • A success signal shows that something ran, not that it worked. Terminated processes, a line in the configuration and a screen full of checkmarks all testify only to a step. Whether the effect has occurred has to be shown by a second route that is independent of that step.
  • An agent’s finding contains measurement and interpretation, and both sound the same. They sit in the same paragraph and in the same tone. Separating them is up to the user, and it costs only one follow-up question: which concrete measurement supports this statement.

None of these cases was solved because the model understood the application particularly well. They were solved because questions that used to be too expensive suddenly cost only a minute, and in one case because a human contributed two observations that no tool shows. The first is what an agent brings when it can query the database and the server directly. The second stays with the human.

FAQ

Where do you start with a vague symptom like “the page is slow”?

Not with a hypothesis but with a comparison. If the same application runs in several environments, the quickest question is whether all of them are affected. If an environment that does not even have the suspect version is affected too, the suspect version is ruled out as the cause, and the search moves to what the environments share. That can be the machine, the network or a shared service.

Why doesn’t docker stats show a memory problem?

Because docker stats only reports the share in RAM, not the swapped-out share. A service whose memory has been swapped out to disk almost entirely shows up there with a very small value. Anyone who knows how much the service normally needs may pause at that, because the value is not inconspicuous but too low. How much has actually been swapped out, however, only becomes visible in the operating system’s memory overview, for example with free -h. The telltale sign is the combination of three things: pages that take seconds on the first request and respond instantly on the second, a low system load, and swap space in use while RAM is free. Each of these alone has other explanations too.

How do you notice that two Docker engines are running in parallel?

The most conspicuous sign is a port that answers although, according to docker ps, the container has no forwarding at all. Add to that data that does not match between the interface and a cross-check, and changes that have disappeared again after reloading the page. The check itself is short: first list the existing Docker contexts with docker context ls and see which one is active and where it points, then determine which process actually holds the port. Both belong before any further container diagnosis, because a wrong context invalidates every measurement that follows.

Should an agent be allowed to run system commands itself?

Within the project, there is little against it as long as the effect can be reversed. System-wide interventions are a different matter, especially when the same machine holds data that does not belong to the project. The division of labor that has proven itself here is that the agent delivers the command sequence together with a pre-check and the human executes it. That costs a minute and keeps the responsibility where it belongs.

How do you stop an agent from making up a cause?

Not completely, because producing plausible text is how it is built. What helps is asking for the measurement that led to the statement. Whoever asks for the concrete query and its return value gets either solid evidence or the information that it is a synthesis. Both are useful as long as they are kept apart. It also helps to have the agent actively look for the opposite, meaning the measurement that would refute its own explanation.

Parent articles:

Sibling articles: