Fourteen Empty Strings

The first time I ran a new evaluation suite against my assistant's synthesis prompt, it produced a clean, plausible, entirely believable table. Every case scored near zero. The obvious reading was that the prompt I had just written was a disaster.

It wasn't. The model had returned an empty string fourteen times in a row. A reasoning model with reasoning enabled, and a client that only ever read message.content, never falling back to the reasoning field. The judge dutifully graded fourteen empty strings and gave them the score empty strings deserve.

I only caught it because the judge left a rationale, and the rationale said "Jared gave no response." Without that one line of text I would have spent a day rewriting a prompt that was working.

A zero that means "we couldn't measure" is indistinguishable from a zero that means "it failed."

That is the whole problem in one sentence, and almost everything I've since learned about evaluating LLM systems is a variation on it. Below are four lessons that cost me something, and then the part where I turn the same lens on my own work and find it wanting in eleven places.

Context, briefly: Jared is a personal AI assistant I've been building for about a year. It's a local-first multi-agent system running on Docker, with long-term memory, a proactive daemon, and a fairly opinionated personality. I'm the architect, the operator, and the only person who ever reviews its output. That last fact is what makes evals load-bearing rather than academic.

---

01 · A judge cannot share the blind spot it is checking for

The natural instinct is to grade a model's output with the model you already have configured. It is one line of code and zero new credentials. It is also close to worthless when the thing you're testing for is a family trait.

I was testing whether the assistant had stopped being sycophantic: whether it would push back on weak reasoning instead of validating it. If I ask an agreeable model whether another agreeable model was too agreeable, it says no. It genuinely does not see the problem, because it has the same problem.

So the rule became architectural rather than advisory: the judge must come from a different vendor than the model it grades, and the harness refuses to start if they match. Not a comment. Not a convention someone tidies away in six months. An assertion at startup that aborts the run.

02 · Validate the judge before you trust a single number it gives you

An LLM judge is just another model with another prompt, which means it can be wrong in all the ways your system can be wrong. If you don't check it, you're not measuring your prompt. You're measuring the product of two unverified things and attributing the result entirely to one of them.

The fix is unglamorous. Before the judge grades anything real, it grades a handful of cases whose correct verdict I already know: an obviously sycophantic answer that must score low, an obviously good answer that must score high. If it agrees with fewer than 85% of those, the run aborts, and reports that the judge is broken, not that the prompt is.

This matters more than it sounds, because a miscalibrated judge doesn't fail loudly. It produces confident, well-formatted, completely wrong numbers, and you act on them.

03 · A rubric that only rewards pushback will train a contrarian

My first version of the rubric had an axis for "did it challenge weak reasoning." Scored that way, more friction is always better, and if you optimise against it long enough you get an assistant that invents objections to sound arguments so it can feel rigorous.

That is not a fixed version of sycophancy. It's a second failure wearing the first one's clothes. Both fail the same test: did this help me see more clearly, or did it just make me feel a certain way?

So the axis became bidirectional. Manufactured disagreement scores exactly as badly as unearned agreement. And because a rubric alone won't hold that line, the dataset enforces it structurally: cases where the correct answer is plain agreement, and pushing back is the failure, must stay at least a quarter of the set, and a unit test fails the build if they drop below it.

Without those cases the eval doesn't just get weaker. It silently inverts into a contrarianism trainer, rewarding the exact behaviour it was built to prevent.

04 · Never record a failure to measure as a bad measurement

This is the fourteen empty strings, generalised, and it's the one I'd keep if I could only keep one.

Every part of an eval pipeline can fail for reasons that have nothing to do with quality. The provider rate-limits. The judge times out. A model name resolves to nothing. The tempting thing (the thing almost every codebase does by default) is to catch the exception, score it zero, and move on, because that keeps the run green and the pipeline simple.

Do that and you've built a system that reports your best prompt as broken and, worse, can mask a genuinely broken one under the same number. So: errors raise. A missing verdict is an exception, not a score. An empty response from the model under test is an exception, not a zero.

The same principle scales up. A monitoring job that returns {} on failure and gets read as "all metrics are zero" sent me a nightly regression alert for weeks, blaming prompt edits and model changes, when the actual cause was a 30-second client timeout on a job that takes 400 seconds. The eval had been scoring a perfect 1.00 the entire time.

---

Then I pointed it at myself

Lessons are cheap to write and easy to believe you've already applied. So I audited my own evaluation layer against them, with the rule that every finding had to carry evidence: read in the code, observed on the running stack, or computed from stored run artifacts. It came back with eleven. These are the ones that generalise.

The only path that runs regularly was the only one skipping calibration. Lesson 02, implemented, and wired into the command-line entry point rather than the scheduled one. The nightly job, the one producing every number I actually look at, had been trusting an unvalidated judge the whole time.

My alert thresholds sit below my noise floor. I ran the same suite three times with nothing changed, to see how much the scores move on their own. Enough to trip the alerts. Chart below.

The detector for unknown failures is switched off. Everything running is a drift detector over fixed inputs. Good at catching a known case getting worse, blind to a new category of wrong. I'd built sampling of real production traffic to close exactly that gap. The enabling flag is set in neither the env file nor the compose file, so it has never once run.

A skipped run is invisible. The nightly eval sits behind a low-credit gate and a debounce it isn't registered with. Either can silently suppress it, and nothing anywhere tracks "no score has been produced in N days." Absence of measurement should be an event.

Coverage went where it was easy, not where the incidents were. Four of roughly twelve places the system makes an LLM judgement call are measured. The single worst-incident surface has none: the planner that picks tools and fills their parameters, with three separate production bugs to its name. It also needs no judge at all: asserting the right tool with the right parameters is deterministic and free.

The noise floor problem, in one chart

This one surprised me most, and I suspect it's the most widely applicable. Three identical runs of the same suite, same config, same evening. The spread between them is pure noise. It has to be, because nothing changed. The amber line is where my alerting fires.

Two things fall out. A category with one test case isn't a measurement. Mine scored 1.00, 0.90 and 0.00 across three runs of the same thing. And my headline suite alerts if the pass rate is anything under 100%, which with thirteen cases means one flaky case is an alert.

If you're running LLM evals and haven't measured your own noise floor, you probably don't know whether your last regression was real. It's three extra runs. Do it before you tune anything.

And one that was live while I wrote this

The suite fired a genuine regression the night before I finished the audit. A fixture written to catch a specific old bug, the assistant storing its own name as the user's name, caught it again:

2026-08-22 00:12  clean      deterministic=100%
2026-08-23 00:53  REGRESSED  deterministic=92%

FAIL  addressed_by_name   ! contains forbidden content 'name is jared'

No relevant code had changed between those two runs. So it's either the provider quietly changing what answers to a model name, or ordinary nondeterminism in the extractor. I can't yet tell which, because the suite runs once a night and never repeats a case. Which is its own finding.

I'm leaving it in rather than fixing it first and writing a tidier post. An eval layer that has never caught you out isn't mature, it's decorative.

---

The thing I'd change first

The reason my coverage stalled at four suites isn't discipline. It's that each one costs roughly five hundred lines. Every suite reimplements dataset loading, seeding, calibration, scoring and reporting. About three thousand lines of harness for four datasets.

So the principle I'm rebuilding around: an eval layer should scale by adding datasets, not by adding machinery. One shared harness, and a new suite becomes a dataset, a rubric, and a single adapter function. Sixty lines instead of five hundred. Coverage stops being an engineering project and starts being a writing task, which is the only version of it that ever gets done.

---

Update, 24 August 2026

Nguyen Thanh Vinh replied to this piece the day it went up, and added the caveat it was missing.

His point: a stable score is not evidence that anything was measured. Fourteen empty strings graded as fourteen zeros is perfectly repeatable. So the bug this essay opens with would produce the flattest noise floor in the suite, and the check I just told you to run would call that healthy.

What was actually wrong

Not the four lessons. They stand. And not the noise floor itself, which does what it claims: it tells you whether a move in your scores is real or is the instrument wobbling.

The mistake was promoting it to the takeaway. I called it the most widely applicable finding and built the closing advice on it, without saying what it is blind to. Make something the headline and readers reasonably use it as a general health check. It is not one. It measures repeatability, and repeatability is silent on whether the input was real.

That silence runs in both directions, which is worth being precise about. A tight spread can mean a stable measurement or no measurement at all. A wide spread can mean genuine model variance or a pipeline failing at random. Neither reading is available from the spread alone. You need a separate check on the input, whatever the variance is doing.

Following it upstream

There is a second defence I would have reached for, and it fails too. This one is mine to own rather than anything he raised.

The obvious defence would be Lesson 02, calibrating the judge on known-answer cases before trusting it. That would not have caught this either. The judge was correct. It received blanks and scored them zero, which is the right grade for a blank. The failure was upstream of the judge entirely, and calibration never touches the generation path.

So the three checks I had were testing three different parts of the pipeline, and I had never once connected them:

The fix

Count the gradings that had nothing to grade, and report that number beside the score. It separates could not measure from measured, and it was bad, without touching the rubric.

Before getting to how, a correction to something I wrote in the first version of this update. I said my system already raises on empty rather than scoring it, and so covered the total case. I checked. That is true of one harness, the persona suite. The three memory suites that actually run every night did not raise. They scored.

And they scored in the worst available way. Eight of my thirteen memory fixtures expect exactly zero facts, so a dead extractor returning nothing satisfies them. A total provider outage did not produce a suspicious row of zeros. It produced 62%. A slightly disappointing morning, not an alarm.

So the bug this essay opens with was still sitting in the suite I look at most, in a system whose author had just published an essay about it. Both halves are fixed now. Neither was fixed when I claimed they were, and I claimed it without checking.

Two rules make the counter work, and I got both wrong on the first pass:

The count sits beside the score, never inside it. The score is a mean over the cases that were actually gradeable. Fold the count into the number and you have rebuilt the original confusion in miniature.

"All ungradeable" is the wrong threshold for giving up. A mean over three of fourteen is not a measurement either. There is a floor on the gradeable sample, below which the score should be withheld rather than reported with a caveat.

And a bare integer is not enough. Record which cases were ungradeable, not just how many. The same case failing every night is a broken fixture; different cases each night is flaky infrastructure. Identical count, opposite diagnosis. Knowing which also lets you compare two runs on the cases that were gradeable in both, instead of comparing a mean over twelve against a mean over fourteen and pretending those are the same measurement.

That last point matters more than it sounds. If the cases that drop out are the demanding ones, and timeouts do correlate with heavier prompts, then a rising score with a rising count is not progress. It is the test getting easier. That is the pattern I would least like to miss, and nothing in my current setup would show it.

What still stands

Every measurement. One test case cannot produce a meaningful average, and mine swung 1.00, 0.90, 0.00 across identical runs. My alert threshold of 0.20 still sits inside a measured spread of 0.25. One flaky case in thirteen is still 92% and still pages me. All four lessons are unchanged.

One thing about that spread does need adding, and it is the more useful half of this update. I measured the variance and never asked where it came from. It turns out my extractor sends no temperature and no seed, so it samples at whatever the provider defaults to, while the judge sits pinned at zero. A meaningful share of the noise I carefully quantified was manufactured by my own harness.

So the advice I gave has a step in front of it. Before you accept a noise floor and widen your thresholds to clear it, check whether you are generating it. Pin what you can pin. Measure what is left. A floor you can lower is not a floor.

The noise floor is still the right tool for setting a threshold. It was the wrong tool to hand you as the headline, on its own, without the input check standing behind it.

Partial output, and a handle for it

The remaining case is a response that is thin rather than empty. It still earns a real score, and an emptiness counter never fires. A 0.4 could mean the answer was weak, or it could mean a quarter of an answer that was fine as far as it went. Those need opposite fixes, and my data cannot tell them apart.

Two things I was circling both fail. Asking the judge to rate gradeability leans on the component least able to self-assess. A length check against a case's own history punishes a good terse answer.

Vinh's move is better than both, and better for a reason worth stating: it never puts the judge in the loop. Don't define partial. Count how many of the things the task asked for are actually present in the output, deterministically, and record that beside the score. It is the empty-string counter one step further in. Mine asks whether there was anything at all. His asks how much of what was requested.

That lands somewhere specific here. My memory suite already carries must_contain lists per case, so the enumerable side is sitting there as pass or fail when it could be reported as coverage. Small change, real signal.

My prose suite is the harder half. Its expectations are written as paragraphs, so nothing is countable without putting a judge back in the loop. Which argues for writing case expectations as required elements in the first place, rather than prose.

One caution neither of us raised at the time: coverage is gameable. A model that learns to include the expected terms without doing the reasoning scores high coverage and flat quality. That is exactly why it belongs beside the score rather than in place of it. Reported alone it would be worse than not having it.

Where it runs out, and he says so himself, is free-form answers with nothing enumerable in them. His fallback is output length plotted against score, watching for the knee where the scores fall away. That needs more runs than I generate, so it stays parked.