Building OsaurusAgent-9b: a local agent model that punches above its weight
How we turned a Qwen 3.5 9B base into a disciplined local agent for tool calling and computer use—without sacrificing general ability.
Osaurus runs language models entirely on your Mac. That means the model powering your agents—the one clicking through settings panes, scheduling tasks, and calling tools on your behalf—has to be small enough to run locally and disciplined enough to trust with your machine. Off-the-shelf models at this size are neither. So we built our own.
This is the story of OsaurusAgent-9b: how we took a Qwen 3.5 9B base and turned it into an agent model that beats a model 30% larger on the agentic work that actually matters inside Osaurus—without losing its general abilities along the way.
Gemma-4-12B-it was evaluated with its native function-calling protocol. No defensible Gemma result was available for OsaurusEvals.
Why 9B
We started from Qwen 3.5 9B, a hybrid-architecture model that mixes linear-attention layers with full attention. The hybrid design is what makes it practical as a local agent: the linear layers keep long agent transcripts cheap to process, while the full-attention layers preserve the precision that tool-call arguments demand. At 9B parameters it fits comfortably on Apple Silicon in quantized form and leaves headroom for everything else your Mac is doing.
The base model is capable but undisciplined as an agent. Out of the box it passed just 56% of our agentic benchmark—it would type into fields without targeting them, fabricate element IDs instead of looking them up, and answer “done” without doing the work.
The method: measure, diagnose, fix—never guess
We did not fine-tune on a big pile of generic agent data and hope. Every training cycle followed the same loop:
- Measure on a held-out benchmark the model has never trained on.
- Diagnose every failure from the raw transcript—not the score, the actual actions the model took.
- Write the hypothesis down before training: which failure classes the next cycle should fix, and the exact pass bar.
- Train a small, surgical adapter on synthetic examples that teach the missing behavior—never the test cases themselves.
- Gate. If the result misses the pre-registered bar—even by one regression—it does not ship. We iterate or roll back.
Two rules kept us honest:
- The benchmark is quarantined. Our 62-case agentic benchmark was written from failure-class analysis, independently of any training corpus, and an automated check aborts any training run whose data overlaps a benchmark case—by ID, by query string, or by UUID. Gains on it measure generalization, not recall.
- Two independent judges. Promotion requires our deterministic grader and an independent LLM review to agree. When one candidate passed the score bar but introduced a single regression, the review’s verdict was blunt: don’t waive a safety criterion under deadline pressure. We iterated until the shipping candidate cleared the bar with zero regressions against the previous certified release, confirmed by a third independent review before anything was quantized.
The failures this loop caught were wonderfully specific. One cycle taught the model that a schedule for “every Sunday at 7:30am” is weekly, not daily. Another taught it that when the harness rejects a bare done action, the right move is to do the work first—not to retry done with a confidently fabricated summary. These are exactly the behaviors that separate a demo from an agent you can leave alone with your calendar.
The results
| Ornith-9b | OsaurusAgent-9b | Gemma-4-12B-it | |
|---|---|---|---|
| OsaurusEvals (official suite, 72 cases) | 79.2% | 90.3% | — |
| Agentic benchmark (62 cases) | 56.5% | 85.5% | 82.3% |
| — tool workflows | 70.0% | 85.0% | 85.0% |
| — computer use | 41.7% | 83.3% | 77.8% |
| — capability honesty | 100% | 100% | 100% |
Against the official OsaurusEvals suite—the decisive test, run on the official harness with independent judge review—the base passes 79% of cases; OsaurusAgent-9b passes 90%. Nine cases fixed, one regressed (an error-recovery probe where it guessed an ID instead of looking it up first). We publish the regression alongside the gains because that’s the deal: every number traces to a raw transcript, including the unflattering ones.
Against a stronger, bigger reference. We benchmarked against Gemma-4-12B-it—a well-regarded public 12B—using its native function-calling, served with its official quantized release. No handicaps: it scored a genuinely strong 82% on our benchmark. OsaurusAgent-9b scores 86%, ahead of a model roughly 30% larger, on a benchmark that is fully quarantined from our training data. The biggest single gap is computer use—the click-type-navigate loop at the heart of Osaurus—where OsaurusAgent-9b leads at 83% vs 78%.
Official OsaurusEvals
The official 72-case suite is run deterministically, with every rubric row re-graded by an independent LLM judge.
| Suite | Ornith-9b | OsaurusAgent-9b |
|---|---|---|
| DefaultAgent (38) | 33 | 37 |
| ComputerUseLoop (23) | 15 | 18 |
| CapabilityClaims (11) | 9 | 10 |
| Total (72) | 57 | 65 |
Agentic benchmark
The held-out 62-case benchmark covers multi-step computer-use loops, tool workflows, capability honesty, invalid-action recovery, type-targeting discipline, mark disambiguation, ID lookup before mutation, and reasoning over tool results.
| Category (cases) | Ornith-9b | OsaurusAgent-9b | Gemma-4-12B-it |
|---|---|---|---|
| Computer-use loops (18) | 6 | 15 | 12 |
| Tool workflows (10) | 6 | 8 | 8 |
| Capability honesty (6) | 6 | 6 | 6 |
| Invalid-action recovery (8) | 5 | 6 | 7 |
| Type-target discipline (5) | 0 | 5 | 5 |
| Mark disambiguation (5) | 4 | 4 | 4 |
| ID lookup before mutation (6) | 4 | 5 | 5 |
| Reasoning over tool results (4) | 4 | 4 | 4 |
| Total (62) | 35 | 53 | 51 |
No lobotomy
Narrow fine-tunes usually pay for agentic gains with general ability. We tracked math (GSM8K), knowledge (MMLU), and instruction following (IFEval) across every cycle, and the shipped model meets or exceeds its base on all of them:
| Retention gate | Ornith-9b | OsaurusAgent-9b |
|---|---|---|
| GSM8K (strict) | 86.0% | 86.0% (tie) |
| MMLU | 78.9% | 79.6% |
| IFEval (prompt-strict / instruction-strict) | 42.5% / 56.6% | 42.5% / 56.9% |
The agentic gains are new capability, not reshuffled capacity: one failure class went 0-for-5 to 5-for-5 on probes written after training ever saw a single example of that behavior.
A note on fair benchmarking
Several of our early reference numbers for Gemma-4 were artifacts—completion protocols it wasn’t designed for, token budgets its thinking overflowed. We threw those runs away and re-ran everything under the protocols each model was built for, because beating a strawman helps nobody. The reference numbers we publish are the strongest we know how to produce. Where we could not produce a defensible number in either direction, such as Gemma-4’s general-retention rows, we publish nothing rather than a weak one.
What it means for Osaurus
OsaurusAgent-9b ships quantized for Apple Silicon (MXFP8 and MXFP4), tuned for the exact tool-calling format and computer-use loop the Osaurus harness speaks: precise element targeting, lookup-before-mutate discipline, honest capability claims, and done actions that carry a truthful account of what actually happened on screen.
It’s the first release in a line. The training loop—measure, diagnose, pre-register, gate—is repeatable, and every remaining failure case on our benchmark is a written-down hypothesis waiting for its cycle.