How monthlet tests AI responses (eval)

This post was machine-translated from the Japanese original published on note. Read the original on note →

monthlet is a note app where you write and grow notes through AI chat. Creating and rewriting notes happens by the AI calling a tool. Ask it to “add a paragraph to the end of my reading notes” and the model picks the append tool and passes a string as its argument. In August we added a harness that machine-scores whether that part behaved as expected — an eval.

After building it and running it a few times, the thing that stood out was that with a harness like this, recognizing what it is not measuring matters more than what it measures. This journal entry is mostly about that side.

Why an app built on AI needs an eval

An ordinary program returns the same output for the same input. Tests are built on that assumption: write down the expected result once, and the next run can be judged the same way.

An app with AI inside is different. Asking the model for the same thing does not guarantee the same result every time. On top of that, the models we use change on the provider’s schedule. Older models get retired, successor generations replace them, and sometimes the name stays the same while the contents are updated underneath. Without changing a single line of our own code, the behavior can be different next month.

So tests aimed only at our own code are not enough. They confirm that the code we wrote is not broken, and that is where they stop. Confirming whether the model behaved as expected needs a separate mechanism.

That mechanism sends real requests to the model, scores what comes back against fixed criteria, and keeps a record. That is the eval. What we want to know is less the pass/fail of any single run and more whether anything changed since the last time we measured.

What the earlier tests missed

Before this, testing had three layers. The first is unit tests for pure functions. The second is a smoke test that actually connects to the three AI providers and checks that the shape of the request is not broken. The third is verification against a running build, driving the same app we distribute from the outside.

What was missing was “did the model behave correctly.” The original smoke test only judges that a final response comes back without an error. It passes if the wrong tool is called, if the note that gets created is thin, or if a help question is answered incorrectly.

monthlet is used with your own API key, so it has to keep up with generational changes in external models. Regressions that break the shape of a request get caught by the smoke test, but there was no layer catching regressions where the behavior degrades.

Of the four layers, only the eval layer was added after the fact.

Four layers of testing: unit tests for pure functions, smoke tests for request shape, E2E tests for the shipped app, and eval for how the model behaved, the one layer added later

Scoring on four axes

There are four axes now.

There are 43 cases. Each one pairs an utterance we wrote with a set of fictional notes to try it against (monthlet calls a note storage location a vault), and each is sent in the same form to Anthropic, OpenAI and Gemini. A full run is 129 API calls; when we ran it in late August it was roughly 2.77 million input tokens and took about six minutes.

Those numbers go stale quickly. Adding cases increases them proportionally, and at one point the README still said 33 cases. When estimating, counting the case definitions again is the reliable way.

The other thing to keep in mind is that the cases are utterances we wrote, not sentences a user actually typed. How hard they are is also a difficulty we chose. The fictional meeting notes include a preamble whose only job is to fill the first 160 characters. Without it, the part that matters does not appear in the list summary, and the question can be answered without reading the note at all. We noticed this on the first run, when all three providers answered correctly without reading.

Every test passed, and a bug shipped anyway

We started with just the A and F axes. Then a bug turned up: the links between notes that the AI writes broke when the filename contained a space. Running the 36 cases we had at the time against all three providers produced zero failures.

The model was calling exactly the tool we expected. What was broken was the string passed in the argument. The A axis looks at the structure of the arguments, not the contents.

That is what the O axis was added for. Write tools are not executed during the eval, but the arguments are already determined, so the thing that would have been created can be inspected without running anything. Three checks: whether the link form is correct, whether ids are being attached to blocks, and whether a type that cannot be rendered is being created.

These checks do not transcribe the correct form into the eval. They import the same functions and constants the implementation uses. If you transcribe, you end up with the implementation fixed and the eval still failing — or the reverse, the eval fixed while the implementation stays old.

The five stages of one exchange and what each axis sees: A covers tool choice and arguments, O the arguments, F and R the reply text. Nothing covers the saved result

The fix landed and the numbers did not move

The link bug itself was fixed a different way: normalize to the correct form immediately before saving.

After that change, running the same cases produced exactly the same results as before the fix. And that is the correct outcome. The O axis looks at the raw arguments the model wrote. A fix applied just before saving does not change the model’s output at all. Only the saved form changes, and that never reaches the O axis.

When we started on the fix, the working note said “now that the O axis exists we can mechanically compare before and after.” We retracted that after running it. Writing down that something is measurable does not make it so; you have to check what the inspection actually looks at before you run it. In the end the fix was covered by two things: a pure function test, and a test confirming that the fixed routine is actually called.

Laid out as a sequence — utterance, tool selection, tool arguments, prose response, save — the four axes each look at a different point. There is still no axis looking at the last one, the saved form.

Count what could not be measured separately from pass and fail

The way we counted the pass rate had its own hole: a number was being produced where nothing had actually been measured.

One day the artifact axis showed 2 passes out of 4. It looks like half failed, but both of the two cases that were measured passed. The other two were transient errors on the API side where no response came back even after retrying. Cases that could not be measured were being counted as failures.

This was not a bug in the newly added axis; it had been that way from the start. Until then, someone wrote an annotation on every past record — “these two are API-side errors, do not read them as an improvement” — to prevent misreading.

Now, unmeasured cases are handled in two buckets.

The opposite failure is closed off too. When there is nothing to inspect — no links at all, zero blocks, an empty response body — the check does not return a pass. Passing there would put us back in the state where every test passed and the bug got through.

Since this split, the per-axis bias became visible. The R axis, which looks at the prose part of the response, has nothing to score when a model calls a tool and emits no prose at all. One of the three providers behaves that way. In the most recent run, that case is counted as a failure marked “could not be inspected,” not “leaked.” Sending to three providers does not mean measuring across three providers.

Also, a red mark on this axis does not mean identifiers are currently shown on screen. They are stripped before display, so what this axis observes is the model’s habit.

Do not compare against a previous run that is not comparable

Because the counting method changed, we stopped showing a delta against records from before that change. Showing it would make an identical result look like a 50-point improvement.

Writing it out, “better than last time” only holds when four things line up: the same case set, the same model, the same counting method, and the same machine. We once subtracted a run scoped to a single case from a full run and got a meaningless number out of it.

For the first three, when conditions differ we show no number and display “no comparable previous run” instead. The fourth, the machine difference, is still undetectable, because run records are not stored in the repository.

Four ways “previous run” breaks: case set, model and counting are stopped, but a machine difference passes straight through

No AI in the scoring

Scoring is limited to things that resolve mechanically to yes or no. We do not have an AI judge the quality of a response.

If the subject under test fluctuates and the grader fluctuates too, a difference from last time cannot be attributed to a change in quality or a change in grading. Swapping the grading model would also break comparability with every earlier number. Format checks let us measure all three providers with the same ruler.

What that leaves unmeasured is correspondingly clear. We do not check whether a help answer is correct as content. We do not check whether a generated note reads well, nor whether the model follows instructions smuggled into a note or a fetched URL. Some of that is covered by other tests, but not as behavior. In the help-lookup cases, the only thing scored is tool selection; the returned body is not inspected, so rewriting the help text does not move the eval numbers.

Not in CI, run by hand

The eval is not in CI. Every run calls external APIs, so instead of wiring it to fire on every change, we run it by hand. The decision to run is based on whether the diff from the previous version touches any of four places: the provider implementations, the prompts, the agent loop, or the tool definitions. One version had a diff of 75 files and 3,113 lines, and zero of those files fell in the four places.

The judgment predicates themselves — the link-form check, the block-type check — are pure functions that call no external API, so they run in the normal test suite on every run. That was added later. They were out of scope at first, and they are exactly the part that produced the wrong number.

Summary

The eval came together in about two weeks, and after that came repeated changes to close gaps in what it was not measuring. The order is the same every time: a bug appears, we notice it sailed through with every test passing, and we add an axis.

So what can be said about this layer is not “we manage AI quality with numbers.” What can be said is this much: the correct tool is called, with the specified argument included, ahead of any forbidden tool; the answer comes back in the specified language; no artifact is created in a form that cannot be rendered; and internal identifiers are not spoken aloud. Whether the content of an answer is correct is still not measured.

Every added axis costs the effort of writing down what that axis does not see. But without that written down, there is no way to decide how to read a result that says everything passed.

Try it

monthlet is currently distributed for macOS (Apple Silicon). A Windows version is in preparation. Enter your own AI API key and all features are available.