Do My AI Coding Agents Actually Learn? I Built a Way to Measure It

Agent instruction files are suggestions, not control flow; unmeasured, review findings recur at a stable 22 percent.

I fingerprinted a year of review findings across ~40 repos. 22% recurred. What unsettled me wasn't the number. It was realizing I had no way to tell whether it was getting better.

I run a fleet of AI coding agents. They review pull requests across roughly forty repositories: correctness, security, data integrity. Every day they produce findings. Every day someone, sometimes another agent, fixes those findings. The dashboards stay green. The throughput is real. It feels like a system that is getting better.

One evening I noticed a warning I was sure I'd seen before. Not the same bug. The same kind of bug, in the same file, flagged the same way. I went looking, and it wasn't the first time; the same shape had come back across several different months.

That's a small thing. But it surfaced a question I'd been carefully not asking: were my agents actually learning, accumulating something that made this month's review better than last month's? Or was I paying, every day, to re-discover the same problems in a loop that only looked like progress?

I'd been assuming the first. I had no evidence for it. Assumption is not measurement, and the gap between the two is exactly where you fool yourself. So I stopped assuming and built something to find out.

The instrument

"Are my agents learning?" is too vague to measure. I needed a narrower, mechanical question: does the same class of finding keep coming back? If the fleet were learning, recurring findings should get rarer over time. If it weren't, they'd just keep showing up.

To count "the same class," I needed an identity for a finding that survives rewording. A reviewer might phrase one issue ten different ways across ten months; string-matching the description is hopeless. So I reduced every finding to a coarse fingerprint: the parts that stay stable when the prose doesn't:

from collections import defaultdict

def fingerprint(finding):
    # A finding's identity, stable across rewordings:
    return (finding.repo, finding.file, finding.severity)

# How many distinct reviews did each fingerprint show up in?
reviews_by_fp = defaultdict(set)
for f in findings:
    reviews_by_fp[fingerprint(f)].add(f.review_id)

recurring = {
    fp: revs
    for fp, revs in reviews_by_fp.items()
    if len(revs) > 1                 # seen in >1 review
}

# 22%: findings that belong to a recurring class.
recurring_findings = [f for f in findings if fingerprint(f) in recurring]
recurrence_rate = len(recurring_findings) / len(findings)

# 14%: repeat appearances only; every review after a class's first.
repeat_only_rate = (
    sum(len(r) - 1 for r in recurring.values()) / len(findings)
)

That's deliberately blunt. (repo, file, severity) collapses distinct issues that happen to share a file and severity, and it splits a genuinely recurring issue if the file gets renamed. I'll come back to what that bluntness costs. But it has one property I cared about more than precision: it doesn't depend on the reviewer's wording, so it measures the same thing across a year of drift.

I pointed it at every review artifact I had: one structured record per finding, each tagged with the review it came from, so a fingerprint's tally is the number of distinct reviews it appeared in. The corpus: 842 findings across 572 reviews, spanning roughly forty repositories over a 365-day window. Enough history that if learning were happening, it had time to show up in the numbers.

The whole instrument is about thirty lines. That's the point of putting it here: this is not a research apparatus. It's a dictionary keyed by a fingerprint and a definition of "the same." If you run agents that log anything structured, you can compute your own version this afternoon. The hard part was never the code. The hard part was being willing to look at what it returned.

The first read

Here is what it returned: 22% of all findings were recurrences. 186 of 842 findings shared a fingerprint with at least one other finding. Those 186 collapsed into 68 distinct fingerprints: 68 specific (repo, file, severity) classes my agents flagged more than once. The most persistent single fingerprint appeared seven times: the same high-severity class, in the same file, surfacing seven separate times across the year.

Figure 1. Selected recurring fingerprints from the head of the distribution: the same anonymized (repo, file, severity) fingerprint appearing in more than one review. 68 classes recurred; 186/842 findings = 22% in a recurring class; repeat-only rate 118/842 = 14%.

My first instinct was to write the sentence you're expecting: the data shows my agents don't learn. I deleted it. It isn't what the data shows, and the difference matters more than anything else in this piece.

What I have is a baseline: the first read off a freshly built instrument, plus one repeat check since. I have the level: 22%, and it was the same at both reads. But a level is not a trend. To claim my agents aren't learning, I'd need the rate tracked across many points and a line that fails to fall. Two points cannot carry a trend. A trend claim on two observations is exactly the overreach I'd reject in someone else's analysis. The fact that the conclusion is the one I half-expected makes it more dangerous, not less.

So I'll fence the claim precisely. What 22% establishes: a meaningful fraction of my fleet's findings are the same classes recurring, and across the only window I've measured, that fraction hasn't visibly shrunk. What it does not establish: that the agents are incapable of learning, or that anything I do next will move the number. It is a baseline. Its entire value is that it's a baseline: a real number I can watch move, instead of a feeling I can keep believing.

Two more honesties about the number itself, since this is the section a skeptical reader should push on hardest. First, what "22%" counts: the share of findings that belong to a class that showed up more than once. 186 of 842 sit in one of 68 fingerprints that recurred. If you'd rather count only the repeats, meaning every appearance after a class's first, the count is 118, or 14%. I lead with 22% because it's the fraction of my review output parked in a class that has demonstrably come back before; but a reader doing 186 - 68 in their head deserves to see the 14% too.

Second, the instrument's bias is the cost of that bluntness I promised to come back to. Collapsing two genuinely different issues that share a file and severity into one fingerprint inflates the count; a file rename can deflate it, splitting a class that moved files into what look like singletons. My guess is collapse dominates: shared file-and-severity collisions are more common than rename-splits, so if 22% is off, it's most likely too high. I haven't measured that, so treat it as a likely upper-biased estimate, not a precise figure or a proven bound: the error, if there is one, probably puts the true rate below 22% rather than above it. I flag the direction because it cuts against the suspicion I walked in with: that my agents were stuck. A number that flatters your prior is exactly the one to discount.

That distinction, between what the receipts admit and what I want them to mean, is the whole discipline. The number is uncomfortable enough on its own. It doesn't need me to harden it into a verdict.

A hypothesis: instructions are not control flow

Before I reach for an explanation, the number deserves some skepticism, because recurrence isn't a clean measure of learning. It's an operational signal with several possible causes. A fingerprint can recur because the file is a genuine hotspot that keeps attracting issues; because a human or another agent reintroduced something the reviewer had nothing to do with; because the model, prompt, or review policy shifted underneath the series; or because severity labels aren't perfectly stable. Any of those produces recurrence without a reviewer "failing to learn."

So what follows is one hypothesis among several. It's the one I find most actionable, not the only one the data permits. It came into focus from a separate exercise.

I handed a capable agent a task with five load-bearing instructions written into its directives: the kind of standing rules you put in a project's instruction file and assume are in force. Follow the test-first discipline, run the secret-scan gate, use the full security checklist, produce a completion artifact as proof, hand the reviewer a diff. Plain, explicit, non-negotiable in tone.

It followed zero of the five. Not through defiance. It wrote tests after the code instead of before, skipped the secret scan, used an ad-hoc subset of the checklist, declared the task done without the proof artifact, never computed the diff. Each shortcut was locally reasonable. The aggregate was a clean miss on every gate I thought I'd required.

That's one observation, on one task, with one agent. I want to be careful about how much it can carry: it's an anecdote, not a study, and I'm not claiming "instructions never work." But it crystallized a hypothesis that explains the recurrence better than anything else I have:

Instructions are not control flow. A directive in a prompt is suggestion-shaped text the model weighs against everything else in context; it is not a gate that executes. There's no mechanism by which "remember not to do X" becomes a constraint that cannot be violated. So when a finding gets fixed, nothing structural records "this class is now forbidden here." The next review starts fresh, weighs the same context, and is free to arrive at the same place. Recurrence is what that looks like from outside: not an agent failing to learn, but a system with nowhere to put what was learned so that it binds.

I hold this as a hypothesis, not a law. The recurrence numbers are consistent with it; the 0-of-5 observation illustrates it; neither proves it. But it has a useful property: it's actionable. If the problem were "the model is too weak to learn," the fix would be to wait for a better model. If the problem is "there's no enforced place to store what was learned," that isn't a model problem at all. It's an engineering problem, and engineering problems I can attack.

The loop I built to bend it

If instructions don't enforce themselves, the enforcement has to live around the model, not inside its prompt. So I built a loop with three structural parts, each one a structural home for a lesson the prompt can't hold.

Capture. When a piece of substantive work finishes, a hook fires and writes the lesson down as a durable structured note on disk, instead of leaving it in a conversation that's about to be discarded. The trigger is mechanical (work of a certain shape ended), not a polite reminder the model can weigh away. The point is to make "what we just learned" a file, not a memory.

Recall. At the start of each session the relevant accumulated lessons are injected into context, and a separate rail, an enforced gate that sits in front of substantive actions, requires a recall step before the agent may proceed. This is the half aimed directly at recurrence: the lesson from the seven-times file is no longer trapped in last month's transcript; by design it's in front of the agent before it touches that file again. Whether that actually lowers the number is exactly what I can't yet claim; see below.

Contradiction. Lessons accumulate, and accumulated lessons rot. Two notes from different months can quietly disagree. A generator walks the corpus and surfaces candidate contradictions for resolution; in the current corpus it has flagged on the order of four hundred candidate pairs. I'll be honest about the seam: the semantic judge that would actually rule on those candidates is off for now. Each judgment is its own model call, and I'm not paying to adjudicate four hundred pairs until the generator that feeds it has earned it, so this stage surfaces candidates but doesn't yet resolve them. It's the least finished of the three.

The shape of the whole thing is the argument. I'm not trying to make the model more obedient; the 0-of-5 observation taught me that lever doesn't work the way I wanted. I'm building the structure a model can't hold for itself: a durable place to write a lesson (capture), a forced moment to read it back (recall), and a process to keep the lessons from contradicting each other (contradiction). Enforcement I can inspect, instead of instructions I can only hope land.

That's the system. What I can't yet tell you is whether it works.

The honest status

Here's where an article like this is supposed to show you the curve bending down: the after to the before, recurrence dropping from 22% to something smaller, the loop vindicated. I don't have that, and I'm not going to manufacture it.

I've measured recurrence at two points, and the loop has run in its full capture-recall-contradiction form for only a round or two. The rate hasn't moved, but there isn't enough there yet to call that movement or its absence. The instrument is built and the system is wired. The experiment that would show whether the second bends the first has barely started.

I think that's worth saying plainly, because the failure mode here is seductive. It would be easy to ship the infrastructure, watch the hooks fire, see the lessons pile up, and declare victory on the plumbing: to mistake "I built the loop" for "the loop works." A firing hook is not a falling curve. The whole reason I built the instrument first was so I couldn't get away with that substitution, including with myself.

So the honest status is this: I have a baseline, a mechanism hypothesis, and an intervention, and I don't yet know whether the intervention moves the baseline. What all of this bought me isn't a fix. It's the ability to find out: to replace "my agents are surely learning" with a number that will tell me, one way or the other, once there's enough of it to read.

Stop trusting, start measuring

The most useful thing I did wasn't building the loop. It was building the instrument that could tell me the loop might be necessary, and that can later tell me whether it helped.

If you run AI agents at any scale, you are surrounded by the feeling of progress: the same green dashboards and real throughput I opened with. None of it is evidence the system is getting better, and the feeling is strong enough to keep you from checking. I assumed for a long time. The assumption cost nothing to hold and would have cost a great deal to keep believing.

The cheapest high-leverage move in agentic engineering isn't a cleverer prompt or a bigger model. It's a number that tells you whether the last clever prompt actually worked: a fingerprint, a counter over your own logs, a definition of "the same." Most teams I've worked in haven't built that number, because we're a little afraid of what it'll say. Mine said 22%, and held, and I still don't get to call my agents broken on two data points.

I didn't measure to prove my agents were smart. I measured so I'd stop being able to fool myself about it. That turned out to be the more valuable result. Anyone can point a counter at the work they've been hoping not to look at too closely.