Surge's New Benchmark Makes AI Agents Do 50 Real Tax Assignments

The benchmark nobody asked for, but everybody needed

Surge published Dayjob: Tax, a benchmark that hands AI agents 50 real tax assignments and grades the output. Not LeetCode. Not HumanEval. Actual professional work: forms, schedules, edge cases, the kind of thing a junior accountant spends a Tuesday on.

The framing is deliberate. Most agent benchmarks measure how well a system writes Python or fixes a merge conflict. Dayjob: Tax asks whether an agent can follow tax rules, apply them to messy inputs, and produce something a professional would sign off on. That shift in the question being asked is the real story here, more than any particular score. For years the industry has treated coding ability as a proxy for general competence, on the theory that if a model can reason through a function it can reason through anything. Dayjob: Tax tests that theory directly, in a domain where the output has to survive review by someone whose signature carries liability.

What the 50 tasks actually test

Surge describes the set as real professional tax assignments rather than synthetic puzzles. That matters because tax work punishes shortcuts: a single misapplied threshold or a missed dependent credit cascades through the return. The mechanism is what makes the domain interesting as an evaluation target. Tax preparation is not a single skill but a chain of them: reading a client’s situation, identifying which rules apply, pulling the right figures from source documents, performing arithmetic that has to reconcile across forms, and then presenting the result in a format that a reviewer or a filing system will accept. An agent can be excellent at four of those links and fail the fifth, and the failure propagates. That is precisely the kind of compounding error that coding benchmarks, with their isolated function-level problems, tend to hide.

Agents have to read instructions, locate the relevant rules, compute, and format. The benchmark scores outcomes, not vibes. Surge has not published a single headline leaderboard number in the post, so treat any “X%” claim you see elsewhere with suspicion until the full results drop. That restraint is worth noting. Benchmark launches often lead with a chart; here the emphasis is on the construction of the task set itself. Readers who want a number to argue about will have to wait, and in the meantime the honest position is that the benchmark’s value lies in its design rather than in any result it has produced so far.

Why coding benchmarks are running out of road

Coding benchmarks have a structural problem: the internet is full of solved code, and models have seen most of it. Tax is different. The rules are written down, but the work is judgment-heavy and the training signal is thinner. This is the contamination argument in its clearest form. When a benchmark’s problems and their solutions are both widely published, a model can score well by having memorized the answer rather than by reasoning to it, and the score stops telling you much about capability on novel inputs. Tax resists that because the underlying statutes and forms are public but the specific fact patterns in the benchmark are not, and because the correct treatment of a given fact pattern depends on judgment about which rules govern.

That makes Dayjob: Tax a cleaner test of whether an agent can operate in a domain where the answer isn’t a Stack Overflow copy-paste. If agents can handle tax, the case for deploying them in legal, insurance, and compliance work gets much stronger. Those fields share the structural features that make tax hard for machines: dense written rules, fact-specific application, high cost of error, and outputs that a human professional must review and stand behind. A benchmark that demonstrates competence in one of them is evidence, though not proof, that the same approach transfers to the others.

The uncomfortable part: grading

Professional-work benchmarks live or die on rubric quality. Tax has defensible answers, but partial credit is where these evaluations get murky. Surge’s reputation is built on human data work, which is presumably why it picked a domain with checkable outputs. The counter-evidence to keep in mind is that checkable is not the same as unambiguous. Two competent preparers can reach different but defensible treatments of the same facts, and a rubric that rewards one and penalizes the other will mislead anyone reading the score as a measure of raw capability. The choice of a domain with verifiable outputs reduces this risk; it does not eliminate it.

The open question is reproducibility. Fifty tasks is small. A single ambiguous prompt can swing a model’s score by a couple of points, and agent scaffolds vary wildly. Watch for whether Surge releases per-task breakdowns and harness details. Without them, a headline number is close to unfalsifiable: you cannot tell whether a strong result reflects the model, the surrounding tooling, or a lucky draw of tasks. With them, other teams can rerun the evaluation, probe the failures, and build on the work. The sample size is the harder problem to solve, since fifty tasks cannot support fine-grained comparisons between closely matched systems, but transparency about method is the minimum a benchmark needs to be useful.

What builders should take from this

If you’re shipping agents into professional workflows, stop optimizing only for code evals. Build a small internal set of real tasks from your own domain, grade them with a rubric a human expert would accept, and rerun it every time you change the model or the scaffold. The advice is deliberately unglamorous. A private set of twenty tasks drawn from your own backlog, with grading criteria written by the person who normally does the work, will tell you more about production readiness than any public leaderboard, because it measures the thing you actually ship. It also gives you a regression test: when a model or scaffold changes, you find out whether the change helped before your customers do.

Dayjob: Tax is a signal that the evaluation conversation is moving from “can it code” to “can it do the job.” The agents that pass the second test are the ones that survive procurement. For readers building or buying agent systems, the practical implication is to start assembling that internal evidence now, rather than waiting for a public benchmark to hand you a verdict on work that only your domain understands.