An eval is a fixed set of test cases with known answers. The same set is run again after every change, so the result shows what the change did.
A pass rate is only worth what its answers are worth: 90% on cases written to have one right answer and 90% agreement with another model are different claims. Each eval below says which one it is.
The benchmark, one line per model
Cases passed out of 36 on the same fictional benchmark, in each state of the app. A bar through a point is the spread of repeat runs. The vertical axis does not start at zero. One case is about 3 points, so a difference of one or two cases is noise.
| State of the app | claude-haiku-5.5 | deepseek-v4.1-flash-1017 | gpt-6.1-terra |
|---|---|---|---|
| Start, 12 Nov | 26.52 runs | 27.52 runs | 27.02 runs |
| Tightened wording (parked)tried, not deployed | 27.02 runs | 26.52 runs | 27.02 runs |
| Degree rule | 26.02 runs-0.5 | 27.52 runs+0.0 | 27.02 runs+0.0 |
| Whole CV | 32.52 runs+6.5 | 36.02 runs+8.5 | 32.02 runs+5.0 |
Oldest first. Only changes that affect the answers are listed.
Code 11 Nov
Pydantic AI agents read the posting, list its requirements and judge each one against the five CV lines that share the most words with it (a TF-IDF word match). The model must cite one line.
Model 12 Nov
Terra, Nova 2 Lite and Nemotron 3.5 were added to the catalog. Without an effort setting a reasoning model picks its own effort, so the cost of a report is a guess; each model row now carries one.
Feature 12 Nov
A second step reads the CV as a whole (what fits, what the CV makes likely, gaps). It never changes the score, and it has its own eval.
Prompt 12 Nov
The explanation under a quote used to repeat the quote. The judge rules now say what the line does not show, in one short sentence.
Prompt 12 Nov
The extractor now treats the first list of expectations as the minimum requirements even when the posting heads it "What you'll do", and keeps the posting's own wording for both lists.
What it did. Afterwards, one live posting on Haiku went from 34% to 30%, and this change was one of the causes. That is one posting, not a benchmark.
Code 12 Nov
OpenRouter rejected the replayed reasoning block ("Invalid signature in thinking block"). Before the fix the overall reading failed in production after a retry.
What it did. Found by reading the production error log; after the fix the overall reading was produced again.
Code 12 Nov
The model is told which lines it may cite for that requirement and tries again. The old downgrade to not shown stays as the last resort.
What it did. Before: Haiku dropped one to four requirements per report. After: 0 of 19 dropped in the next Haiku report (one report).
Model 12 Nov
Muse Ember (and, a day later, Qwen) failed every request with "tool_choice required is not supported". Their catalog rows now use prompted JSON output.
Baseline 12 Nov
Judge rules as they were after the reason-wording change, five word-matched CV lines per requirement, no rule for degrees with an equivalent. Rebuilt for the benchmark by putting the later settings back; see evals/history_prompts.json.
On the benchmark. claude-haiku-5.5 26.5 of 36; deepseek-v4.1-flash-1017 27.5 of 36; gpt-6.1-terra 27.0 of 36.
Measurement 13 Nov
Not a change to the app: a measurement. It showed that the 8192-token answer limit made reasoning models fail on 4 to 24 of 38 runs, that Haiku and Terra were unaffected, and that models split into a strict and a generous group.
What it did. Details in docs/MODEL-COMPARISON-2026-11-13.md.
Prompt 13 Nov tried, not deployed
The judge was told that seniority, a neighbouring field or merely transferable skills are not a part of the requirement. Measured on 16 postings it moved Terra and left Haiku, the live model, where it was, so it was not deployed.
What it did. On 16 real postings Terra answered "none" on 50% of known gaps instead of 30%, and Haiku stayed at 46%. On the benchmark it changed nothing that stands out (Haiku 26.5 to 27, DeepSeek 27.5 to 26.5, Terra 27 to 27 cases).
On the benchmark. claude-haiku-5.5 27.0 of 36; deepseek-v4.1-flash-1017 26.5 of 36; gpt-6.1-terra 27.0 of 36.
Prompt 13 Nov
"Or equivalent experience" is met by about ten or more years in the field. The CV's profile line, which usually states the years, is always among the lines offered for such a requirement.
What it did. Found from a real chat: the report said a degree was not shown although the CV states fifteen years in the field. On the benchmark it did not help: the long-career degree case still failed on all three models, because the benchmark CV has no profile heading and the fallback only looked for years written as digits or ten, fifteen, twenty. Reading the whole CV fixed that case (see the next entries).
On the benchmark. claude-haiku-5.5 26.0 of 36 (-0.5); deepseek-v4.1-flash-1017 27.5 of 36 (+0.0); gpt-6.1-terra 27.0 of 36 (+0.0).
Code 13 Nov
The judge now receives the entire CV once and may cite any line for any requirement. The five-line word match could hide a line that shares no words with the requirement ("win-back email series" for churn).
What it did. On 16 real postings with Haiku: mean score 54.7 to 65.8, cost per report 0.75 to 0.52 cents, no extra credit on lines the owner's notes call gaps. On the benchmark: the cases where the deciding line shares no word with the requirement went from 0 of 36 answers right (three models, two runs each) to 28 of 36. With word matching the model was never shown the line.
On the benchmark. claude-haiku-5.5 32.5 of 36 (+6.5); deepseek-v4.1-flash-1017 36.0 of 36 (+8.5); gpt-6.1-terra 32.0 of 36 (+5.0).
Code 13 Nov
A longer CV fell back to the word-matched lines before. Now every CV a visitor may upload is read whole.
What it did. A CV padded to 52,000 characters on Haiku: score 65.0 against 65.8, 0.94 against 0.52 cents per report, no failures.
Prompt 13 Nov
The red label read as a warning against the applicant. The stored verdict is unchanged; the words and the label colour are not.
What it did. Two real Haiku runs afterwards still produced their overall readings. Not measured on the benchmark.
Feature 13 Nov
Reviewing a report no longer stops at the verdict: the cited line can be changed. The citation eval keeps checking the model's own line.
With one example each
Asks: Does the whole check work on tiny cases where the right answer is obvious?
Cannot tell you: Easy on purpose. Models that differ by 20 points on real postings can all score 90% here, which is why the benchmark exists.
Asks: Does a model agree with the verdicts the owner corrected by hand?
Cannot tell you: Only as good as the reviews, and it needs about 30 reviewed items before the figure means anything. It uses real reports, so it is not part of the public demo.
Asks: Is the CV line quoted under a verdict really evidence for it?
Cannot tell you: A second model does the judging, so the number is agreement with that model, not truth.
Asks: Does the chat refuse what it must refuse and answer what it must answer?
Cannot tell you: Rules checked by code on fixed questions. It does not judge whether an answer is right, only whether it stays in bounds.
Asks: Does the overall reading follow its own rules?
Cannot tell you: Three fixed postings against a small hand-written CV. It checks the rules, not whether the reading is wise.
Asks: Does the judge call a requirement met only when a CV line shows it, and does it find that line when the words differ?
Cannot tell you: Thirty-six fictional cases, so one case is about 3 points: a difference of one or two cases is noise. It tests the judge, not the quality of the letter or the chat.
| Model | reverse-apply-benchsynthetic |
|---|---|
| claude-haiku-5.5 | 89%36 cases |
| deepseek-v4.1-flash-1017 | 100%36 cases |
| gpt-6.1-terra | 89%36 cases |
| Started | Dataset | Model | Cases | Passed | Status |
|---|---|---|---|---|---|
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of whole-cv | gpt-6.1-terra openrouter | 36 | 89% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of whole-cv | deepseek-v4.1-flash-1017 openrouter | 36 | 100% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of whole-cv | claude-haiku-5.5 openrouter | 36 | 89% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of whole-cv | gpt-6.1-terra openrouter | 36 | 89% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of whole-cv | deepseek-v4.1-flash-1017 openrouter | 36 | 100% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of whole-cv | claude-haiku-5.5 openrouter | 36 | 92% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of degree-rule | gpt-6.1-terra openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of degree-rule | deepseek-v4.1-flash-1017 openrouter | 36 | 78% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of degree-rule | claude-haiku-5.5 openrouter | 36 | 72% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of degree-rule | gpt-6.1-terra openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of degree-rule | deepseek-v4.1-flash-1017 openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of degree-rule | claude-haiku-5.5 openrouter | 36 | 72% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of tightened | gpt-6.1-terra openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of tightened | deepseek-v4.1-flash-1017 openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of tightened | claude-haiku-5.5 openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of tightened | gpt-6.1-terra openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of tightened | deepseek-v4.1-flash-1017 openrouter | 36 | 72% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of tightened | claude-haiku-5.5 openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of start | gpt-6.1-terra openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of start | deepseek-v4.1-flash-1017 openrouter | 36 | 78% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of start | claude-haiku-5.5 openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of start | gpt-6.1-terra openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of start | deepseek-v4.1-flash-1017 openrouter | 36 | 75% | Done |
| 13 Nov, 16:01 | reverse-apply-benchsynthetic, as of start | claude-haiku-5.5 openrouter | 36 | 72% | Done |