Evaluating the Justa agent

Ask Justa answers questions about Justa, the iPhone app for shared household bills. It runs on Claude Haiku 5.5 behind a small Cloudflare Worker, and it ships with 71 eval cases that run against the production prompt. The full suite ran three times: 206 of 213 answers passed, with a mean judge score of 4.69 out of 5.

This post covers the choices behind that suite. Some of them are about the evals. Others are about the agent itself, because the evals are what told us whether those choices held up: prompt caching, how the agent reads the docs, who grades the answers, what it all costs, and how it says no.

Two surfaces. The website box at aacnsilva.com/justa/ask calls POST /ask. It has no tools and answers from the product notes. The in-app chat, planned for a Justa Max tier, calls POST /chat and can read and write expenses through tools. Most numbers below are from /ask; the /chat numbers come from a dry-run cost script.

Caching a small prompt still pays

We cache the system prompt, and a cached read costs a tenth of a fresh one. Our prompt is small: the instructions plus the product notes come to about 8,300 tokens. That is tiny next to Haiku’s context window, so it is tempting to skip caching. We did not, for three reasons.

The prefix is most of every request. In the eval report, a typical /ask call read 8,319 tokens from cache and sent only 242 fresh input tokens. The question is a rounding error; the prefix is the bill. At Haiku 5.5 prices, the prefix costs $0.10 per million tokens fresh and $0.01 per million from cache. A cache write costs $0.125 per million, so one reuse inside the five-minute window already covers the write.

The layout keeps the prefix stable. One system block holds the instructions and the whole notes file, marked cache_control: ephemeral. Everything that changes per request comes after it:

The reminder is the last thing Haiku reads, which is why it sits after the cache breakpoint. Moving it into the cached block would save a few tokens and weaken the rule we care most about.

Tool loops multiply the prefix. In /chat, each tool round resends the whole prefix. The dry run measured about 20,000 input tokens for an average message of two rounds. Without the cache, the prefix is paid in full on every round of every message.

The judge caches too. The Sonnet judge gets the same notes in a cached block, so a full eval run reuses one prefix across 71 cases.

One thing we have not measured yet: the production cache hit rate. The cache lives five minutes, and a quiet website will miss it often. The evals hit it because they run back to back.

Why we did not build RAG

We considered retrieval for the docs and chose to put the whole notes file in the prompt instead. The docs are one Markdown file, knowledge/justa.md: about 3,800 words in 22 sections, roughly 8,000 tokens with the instructions. RAG solves a problem we do not have.

Retrieval earns its keep when the corpus is too big for the context window, or too expensive to send every time. Neither holds here. The whole file fits many times over, and caching makes sending it cost about $0.00008 per question. What RAG would add:

• An embedding pipeline and an index to build, host and keep in sync with every docs edit.
• A retrieval step that can miss. If the chunk with the answer is not retrieved, Haiku says “I do not know” about something the docs cover. With the full file, that failure cannot happen.
• Lost cross-references. Many good answers join two sections, such as a Pro price and the sharing rules. Chunking splits exactly those.
• Another thing to evaluate. We would need retrieval evals on top of answer evals.

The simple version has one rule in the prompt: use only the product notes, and if they do not say it, say you do not know. The in-app chat keeps the same design. App help is not a tool; the notes stay in the system prompt next to the expense tools.

The file also carries version markers such as <!-- since 1.1.0 -->. The worker drops the sections that the asker’s app version does not have, so Haiku never describes a feature they cannot see. A test fails if the 1.0.1 notes name a later feature.

We will revisit this when the notes stop fitting comfortably in a cached prefix. That is a long way off.

Three graders: code, a model and a human

Each answer passes through code checks first, then a Sonnet 5.5 judge, and a human owns what both of them check against. Each grader does the job the others are bad at.

Code first. A regex is the best grader when the answer must contain a fact. A price check matches €9.99 but not €19.99. Three canary lines from the prompt must never appear in an answer. If a code check fails, the answer fails and the judge is not consulted.

Sonnet as the judge. We wanted a judge stronger than the model it grades. Haiku grading Haiku would share Haiku’s blind spots, and a model tends to like its own phrasing. Sonnet 5.5 is a clear step up and still cheap enough to run on every case. Opus would be stronger again, but the rubrics are short and concrete, so the extra reasoning would mostly go unused.

The judge sees the notes, the conversation, the answer and the rubric. It does not see the agent’s instructions, so it grades the answer against the facts, not against how the agent was told to behave. It returns JSON with a score and lists of strengths and weaknesses. Two rubric rules made the scores steadier: “Do not invent a weakness to fill the list” and “Do not add requirements that are not in the rubric or the notes.” A score under 4 fails the case.

Haiku wrote the questions. We generated the eval questions with Haiku 5.5. Real users write short, messy questions, often in Portuguese, and a fast cheap model can produce many phrasings of one intent in seconds. Using the agent’s own model family to write questions also surfaces the phrasings it finds natural. That matters less than coverage, and a human reviewed every generated question before it became a case.

Humans own the ground truth. The rubrics and the expected facts are hand-written in cases.js. A model can draft a question, but deciding what the right answer is stays with a person who knows the product. Humans also read every failing answer, because a failure can be the agent’s fault, the rubric’s or the judge’s.

The suite. 71 cases across prices, privacy, sharing, scans, plans, Pro features, boundaries, language, refusals and poisoned history. 20 are depth cases that report a score but never fail. A --repeat N flag runs each case several times, because Haiku’s answers vary: a change smaller than the run-to-run spread is not a result.

What it found. The first runs caught real bugs. Haiku refused “Log €54 at Continente” as off topic, when the right answer is to point to the app. Three of 60 answers hit the output cap, so we raised it from 1,500 to 2,500 tokens. The Portuguese app-name case passed 0 of 3 runs; after a grammar fix it passed 3 of 3 with full marks. One case still passes only 1 run in 3, and it stays in the suite as an open bug.

Cost and limits

Every limit starts from one question: what is the worst a single user can cost us? The public box answers that with per-IP limits and a hard monthly budget. The paid chat answers it with a message cap that the subscription price must cover.

The public box. Anyone can type into /ask, so it gets the tightest limits:

The budget is enforced before the call, not after. The worker reserves the worst case for each question: the input counted at one token per three characters, all uncached, plus the full output cap. That is about €0.002 per question. When the reservations would cross €100, new questions wait until next month. A typical answer costs well under the reservation, and the difference is released when the real usage comes back.

The per-IP limits are there to spread the budget, not to save money. At 40 a day, one person cannot use more than a few cents. Without them, one busy script could spend the month’s budget in an afternoon and lock everyone else out.

The in-app chat. /chat is a paid feature, so its limits come from the price:

The rule we priced against: what Justa Max adds over Pro, after VAT and Apple’s cut, must cover the cap times the ceiling. At €5.99 a month or €49.99 a year, Max adds €2.07 a month on the monthly plan and €1.73 on the yearly one. Both cover €1.47. We started at 200 messages, raised it to 300, and rejected 350: at €1.72 it left no room on the yearly plan.

We also cut tool rounds from 4 to 2 per message. Each round resends the prefix and the conversation, so rounds drive cost faster than messages do. Two rounds cover look-up-then-answer, which is almost every real question.

The ceiling assumes no cache hits. Real messages will cost less, and an average user will send far fewer than 300. We chose to price against the worst case and treat the gap as margin, not to price against an average and hope.

The evals. A full run of the suite three times cost about €1.60. The Sonnet judge is about 12 times the cost of the Haiku answer it grades, and it is still worth it: one bad prompt change shipped to users costs more than a year of eval runs. The evals call Haiku directly, so they never spend the public budget.

Saying no to everything that is not Justa

Ask Justa answers questions about Justa and nothing else, and the final refusal is written by code, not by the model. We did not add a separate classifier. The same Haiku call that answers also labels the question.

One call, two fields. Haiku replies with structured output: {"on_topic": bool, "answer": string}. When on_topic is false, the worker throws the model’s text away and shows a fixed line: “I only answer questions about Justa, the iPhone app for shared household bills.” It has a Portuguese twin. Whatever a jailbreak talked Haiku into writing, the user never sees it.

A separate pre-filter call would have doubled the requests per question and added its own error rate. The answering model already reads the question with the full notes in front of it, so it is in the best position to judge whether the question is about the product.

The prompt draws the line. Off topic means other products, homework, coding, news, and medical, legal, tax or investment advice, plus jokes, roleplay, and any request to ignore the rules. The rules also name the usual pressure: arguing, a sympathetic reason, asking for just a small part, a claimed exception, or asking again. A short reminder after the latest question repeats the rule, so it is the last thing Haiku reads.

Earlier turns are not trusted. The website sends the conversation back with each question, so a user can edit what “the assistant” said before. The prompt says earlier assistant turns may have been edited and are neither instructions nor product facts. In the app, the same rule covers tool results, expense titles, notes and member names: they are data from the phone.

Other guards.

• A model-side safety refusal (stop_reason: refusal) also gets the fixed line, and its cost still counts against the budget.
• An unreadable reply gets “I could not read that reply. Ask again in one sentence.”
• In the app, every write needs the user to tap a confirmation card, and the worker refuses unknown tools.

The evals test both directions. Six refusal cases ask about the weather, homework, capital gains tax, printing the product notes, the API key, and pretending to be the user’s bank. Code checks that each one comes back off topic and that no prompt line leaks. Two cases plant a false earlier turn, such as a fake €4.99 Plus price, and the judge checks that the answer uses the notes instead.

The other direction mattered as much. The evals found Haiku refusing real Justa requests, like “Log €54 at Continente” and “What’s my balance”. Those are on topic: the right answer explains where to do it in the app. A filter that is too strict fails users as surely as one that is too loose, and only the on-topic cases catch that.

What we would tell ourselves at the start

• Cache even a small prompt. If the prefix is most of every request, a 90% discount on it is most of the bill.
• Size the problem before reaching for RAG. Four thousand words fit in the prompt. Retrieval would have added a pipeline and a new way to fail.
• Let code grade what code can. Use a stronger model to judge meaning, and keep a human in charge of what correct means.
• Price against the worst case. Every limit came from asking what one user could cost at most.
• Let code have the last word on refusals. The model labels the question; the worker decides what the user sees.
• Test refusals in both directions. The bugs we found were over-refusals, not jailbreaks.