Mistral's 12B Model Beats GPT-6.1 on SQL—But There's a Catch

A 12B model just posted a SQL score that makes frontier labs nervous

Mistral released a 12-billion-parameter model this week called SQLMistral-12B, and the headline number is 94.2% exact-match on the Spider 2.0 dev set. For context, OpenAI’s GPT-6.1 Terra scored 93.8% on the same benchmark in its own model card. A model roughly one-fifteenth the size of a frontier dense model just edged it out on text-to-SQL.

That is the kind of number that gets screenshotted and posted to every Slack channel in a data engineering org. But before you rip out your warehouse copilot, read the evaluation appendix. Mistral’s own footnotes say the 94.2% was measured with a constrained decoding harness that rejects any query not matching the provided schema’s grammar. GPT-6.1’s 93.8% was reported under unconstrained generation. That is not an apples-to-apples comparison, and Mistral knows it.

What Spider 2.0 actually measures

Spider 2.0 is the harder successor to the original Spider benchmark. It includes 632 dev examples across 178 databases, with schemas ranging from 4 to 300+ columns. The benchmark’s own maintainers at Yale published a note in August 2026 warning that exact-match scores vary by up to 6 points depending on whether the evaluation uses execution accuracy or exact string match, and whether foreign-key hints are provided in the prompt.

Mistral’s model card lists “execution accuracy” for the 94.2% figure, not exact match. OpenAI’s 93.8% for GPT-6.1 Terra is listed as “exact match” on the same page. Execution accuracy is generally easier because a query can be syntactically different but return the right rows. So the two numbers are not measuring the same thing. Mistral’s PR team did not mention this in the launch thread.

To be fair, Mistral also reports 91.7% execution accuracy for GPT-6.1 Terra when re-run under the same harness. That number is in a footnote on page 14 of the technical report. So the real gap, when measured consistently, is 94.2% vs 91.7%—a 2.5-point win for the 12B model. That is still impressive, but it is not the “beats GPT-6.1” headline.

The constrained decoding trick

The harness Mistral used is not cheating, but it is doing heavy lifting. Constrained decoding forces the model to emit only tokens that keep the SQL parse tree valid. This eliminates a whole class of errors—unclosed parentheses, misspelled column names, invalid JOIN syntax—that unconstrained models produce regularly.

In practice, this means SQLMistral-12B is optimized for a specific deployment pattern: you give it a schema, you run it behind a grammar-aware inference server, and you get clean SQL. If you drop it into a generic chat interface with no grammar constraints, its score drops to 88.4% on the same dev set, according to Mistral’s own ablation table. That is still better than most 12B models, but it is no longer beating GPT-6.1.

The lesson for builders is straightforward: benchmark numbers are only meaningful if you know the inference harness. A model that wins on a constrained decoder may lose in your RAG pipeline if you are not using the same constraints.

Pricing and availability

SQLMistral-12B is available under Apache 2.0 on Hugging Face. Mistral’s hosted API charges $0.14 per million input tokens and $0.42 per million output tokens, which undercuts GPT-6.1 Terra’s $2.50/$10.00 by a wide margin. For a 12B model, that pricing is aggressive but not shocking—Mistral has been pricing small models as loss leaders for two years.

The more interesting detail is the context window: 32,768 tokens. That is enough for most single-database schemas but not enough for the multi-database queries that Spider 2.0 includes. Mistral’s own report notes that 14% of the dev set exceeds 32K tokens when the full schema is included, and those examples were truncated. Truncation tends to hurt accuracy, so the 94.2% is actually measured on a slightly easier subset than the full benchmark.

What the fine print hides

Three caveats matter for anyone evaluating this model for production:

1. Schema complexity: Mistral’s report breaks down accuracy by schema size. On databases with fewer than 20 columns, SQLMistral-12B scores 96.1%. On databases with more than 100 columns, it drops to 81.3%. GPT-6.1 Terra scores 89.7% on the same large-schema subset. So the 12B model wins on small schemas and loses badly on large ones—which is exactly the opposite of what most enterprise data warehouses need.

2. Dialect coverage: The model was fine-tuned primarily on PostgreSQL and SQLite. On Snowflake and BigQuery dialects, Mistral reports a 4-6 point drop in execution accuracy. GPT-6.1 Terra was trained on a broader dialect mix and shows less degradation.

3. Few-shot sensitivity: SQLMistral-12B’s accuracy swings by up to 9 points depending on how many few-shot examples are in the prompt. Mistral’s headline number uses 5-shot prompting. With zero-shot, the model scores 79.8%. That is a steep cliff, and it means you need a retrieval system to pick good examples if you want the advertised performance.

None of these caveats make the model bad. They make the headline misleading. A 12B model that scores 94.2% under ideal conditions is still a remarkable piece of engineering, and for many builders the price-performance tradeoff will be worth it. But the claim that it “beats GPT-6.1” is only true under a specific harness, on a specific subset, with a specific prompt format.

The bigger picture: small models are eating benchmarks

This is the third time in six months that a sub-15B model has posted a headline number within 2 points of a frontier model on a narrow benchmark. In August, Qwen’s 14B model matched GPT-6.1 on HumanEval. In September, a fine-tuned Llama 3.3 8B beat it on MMLU-Pro’s law subset. Each time, the fine print revealed a harness difference or a subset restriction.

The pattern is clear: narrow benchmarks are saturating, and the gap between small and large models on specific tasks is shrinking. That is good news for builders who want cheap, fast, locally deployable models. It is bad news for anyone who reads benchmark tables as if they were apples-to-apples comparisons.

The right way to evaluate SQLMistral-12B is to run it on your own schema, with your own dialect, using the harness you actually plan to deploy. Mistral’s numbers are a starting point, not a verdict. If your warehouse has 200-column tables and you query Snowflake, the 12B model may not beat GPT-6.1 at all. If you have a clean PostgreSQL schema and you can afford to run a grammar-constrained decoder, it probably will—at a fraction of the cost.

What to watch next

Mistral has not announced a larger SQL-specialized model, but the technical report hints at a 22B variant in training. If that model closes the large-schema gap while keeping the price low, it could become the default choice for text-to-SQL pipelines. For now, SQLMistral-12B is a strong option for small-schema, PostgreSQL-heavy workloads—and a cautionary tale about reading benchmark headlines without the footnotes.