Accounted Bench
How good are language models at real accounting work?
Why this benchmark exists, and what it measures that others do not.
Swedish bookkeeping is not a matter of taste. Bokföringslagen fixes what a verifikat must contain, when a business event must be recorded, and how a mistake may be corrected; mervärdesskattelagen fixes which of ten VAT treatments applies and which ruta it lands in. An accounting agent that is fluent but wrong is worse than no agent, because its output enters a legally binding record that a human later signs.
That makes the domain unusually testable. The correct end state of a booking is decidable, not a matter of preference, and Accounted's production database already enforces the law in triggers: an entry that does not balance, an edit to a posted voucher, a write into a locked period. Those triggers are the oracle. Where other agent benchmarks grade a transcript or ask a model to judge a model, this one asks a database whose constraints are the statute.
That also fixes the scope of a single board. The oracle is national: the chart of accounts, the VAT act and the bookkeeping act are Swedish, so a model that books reverse charge correctly here has not been shown to know Norwegian mva or German Umsatzsteuer. Accounted Bench is therefore built one jurisdiction at a time, and Sweden is the first. The other countries on the selector below are named so the scope of this page is legible, not because a number exists for them: not one of them has a task written, and not one has been run.
Accuracy is not the whole question. A model that books at 96% still cannot be trusted unattended unless it can tell you which 4% to check, so every task also elicits the model's own confidence, and the board reports how much a model could commit alone while staying right. That figure is measured out of sample, with the threshold fitted on tasks it is not then scored on. At this suite's size the honest version of that metric turns out to track raw accuracy closely: the models we would let book alone are simply the most accurate ones. Whether calibration earns a model more than its accuracy already does is a question a larger suite could answer and this one cannot yet.
One thing to fix in your head before reading the board. The numbers here are higher than published accounting benchmarks report, and that is a property of the question, not of the models: APEX-Accounting asks nine frontier models to reconcile, accrue, post and report across ten environments and its best score is 56.4%; this suite asks for one account and one VAT treatment on one outgoing expense against one chart of accounts. A narrower question earns a higher number. It does not earn a broader claim, and nothing on this page should be read as "models can keep books".
Four suites, four separate rankings, deliberately unblended: a model that reads receipts well but books reverse charge wrong is exactly the distinction an operator needs to see. Booking opens on strict (the single gold account and the right VAT), because the lenient rate beside it is biased upward by construction. The lenient ranking, and with it the statistical tie groups, is one click away in the legend.
The booking tasks where models most disagree, exactly as posed, with every model's actual answer.
Pick any two models: per-suite results, run cost, unique solves and shared misses.
The booking tasks and their answers were authored with Claude, and Claude models are then ranked on them. That is the sharpest objection to this board, and it is addressed by the two checks below, each a re-runnable script in the repo. Neither is a substitute for review by practising accountants, which the gold has not yet had: a model auditing a model shares a model's blind spots, and a bias test rules out one mechanism rather than the whole objection. Read the correction log under them as what the process has caught so far, not as a guarantee.
What the audit changed
The agreement rate above is measured on the gold as the auditors saw it, before these changes, so it is the score the process earned rather than the score it was left with. Every disagreement that survived review is listed; the ones omitted proposed an account the task already accepted, so they cost the answering model nothing.
Every task, every raw run record and the source of this page are public in the Accounted Bench repository: tasks in tasks/, one JSON line per model call in results/runs/, the scoring in src/aggregate.ts, and the full methodology in README.md. Tasks and results are CC BY 4.0.
The strongest objections we know of, stated by us rather than left for someone else to find.
v2.1 measures models. The axes that follow measure everything else in the sentence "this model, with these instructions, in this system".
A coverage profile of the published 52 found them to be expenses without exception: no revenue, no payroll, no tax account, no financial items, no foreign currency, and one single company that is not VAT-registered. Twenty-five tasks covering everything but revenue are written and reviewed, and wait on a full run across all fifteen models before they join the board. Revenue needs a decision first, because the VAT treatment enum has no output-VAT value, so sales either force a change that invalidates the current board or belong in a suite of their own.
Sweden is the first, not the only one worth measuring. The jurisdictions on the selector are the queue behind it, the Nordics and the Baltics first, then the larger EU markets and the UK. Nothing has been run for any of them: each needs its own chart of accounts, its own VAT act, native documents and a ledger whose triggers enforce that country's bookkeeping law. Until that exists the selector shows what is missing rather than a placeholder score, because a benchmark whose oracle is the law cannot be ported by translating the prompts.
The same model, given different instruction sets, is a different bookkeeper. The next axis holds the model fixed and varies the skill: a bare prompt, a domain skill with the BAS chart and VAT rules loaded, a full agent briefing. What a skill is worth, in points and in kronor, is measurable.
Accounted Bench deliberately uses one minimal scaffold so models are comparable. That makes the scaffold itself an untested variable. Running the same tasks through retrieval-backed, self-consistency and staged-approval harnesses says how much of the remaining error is the model and how much is the plumbing around it.
The end state is a sentence a buyer can act on: this model books at X% inside Accounted, and at Y% inside another Swedish ledger. Tool surfaces, account defaults and posting rules differ between systems, and that difference is a property of the system, not of the model. Measuring it is the point of owning the benchmark.
Planned, not measured. Nothing in this section has been run; it is published so the direction is on the record before the numbers exist.