ClickHouse just published a benchmark on "data agents": AI systems meant to answer analytical questions in plain language, directly against a data warehouse. I posted about it on LinkedIn, summarizing the numbers that caught my attention. Checking the full source for this article, my summary held up, but the benchmark has a lot more useful nuance than what I'd picked up skimming it.
The number that stopped me: on the same 201 questions, the most accurate model (Claude Fable 5.1, 76.6% correct) costs $52 to run across the whole benchmark. An alternative, DeepSeek V4 Flash, scores 65.7%, 11 points lower, for $1. Fifty-two times cheaper for eleven points less accuracy.
Should you pick the top model on the leaderboard, or the one that actually fits your cost and volume constraints? Once you look at how the benchmark is built, it answers that question indirectly: it depends far less on the model than you'd think, and far more on what you give it to understand before it writes a single query.
What the benchmark actually measures
Classic text-to-SQL benchmarks (Spider, Spider 2.0, WikiSQL, BIRD) measure one thing: a model's ability to translate a natural-language question into a correct SQL query. That's useful, but it isolates a single step out of a problem with a lot more steps when you're talking about a real analytical agent.
The ClickHouse benchmark tests the full pipeline: understanding the warehouse schema, picking the right tables, building a coherent query plan, generating the SQL, executing it, then interpreting the result to produce a natural-language answer. Any of these steps can sink the final answer, independently of how good the generated SQL is.
One methodological detail changes everything: ClickHouse specifically controlled for memorization. Frontier models can complete questions from well-known benchmarks from training memory rather than actually reasoning through them. ClickHouse shows that if you give a model the first 60% of a Spider question and ask it to guess the rest, the best "memorizer" completes the text with 0.89 word-level similarity, nearly verbatim. A public benchmark used for years ends up measuring memorization instead of generalization.
To avoid that, the 201 questions in this benchmark come from DWAINE's real production traffic, ClickHouse's internal analytics agent (which handles about 70% of queries on their data warehouse, for over 200 internal users, roughly 33 million LLM tokens a day), and the warehouse the models are tested against is a synthetic but faithful reconstruction of the real one, with entities generated by Faker instead of real organization names. Asking for a specific customer's ID can't be answered from a model's training memory, only by actually querying the data.
The results, in detail
28 models were tested, across every vendor and type (proprietary and open-weight), with the same agent harness, the same prompt, the same warehouse.
| Model | Accuracy | Cost (201 questions) |
|---|---|---|
| Claude Fable 5.1 | 76.6% | $52 |
| DeepSeek V4 Pro | 74.6% (-2 points vs Fable 5.1) | over 75% cheaper than Fable 5.1 |
| DeepSeek V4 Flash | 65.7% (-11 points vs Fable 5.1) | $1 (52x cheaper than Fable 5.1) |
The top ten spots on the leaderboard are all frontier models, five of them open-weight, when ranking on accuracy alone. But the moment you factor in cost, the ranking changes completely: DeepSeek V4 Pro sits only two points behind the leader for over 75% in savings, which makes it a far more reasonable pick than the raw leader for most real-world use cases.
The real bottleneck: planning errors
This is the finding that struck me most, and the one my LinkedIn summary didn't dig into enough. ClickHouse classifies every failure into five failure modes:
- FM1, no attempt: the model gives up before producing an answer. Rare at the top of the leaderboard, but it climbs fast further down: 34% for Gemini 2.5 Flash, 27% for Gemini 2.5 Pro, 23% for Qwen3-Coder 30B.
- FM2, wrong plan: the model picks the wrong tables, or builds a query plan that doesn't actually answer the question, before it even writes any SQL.
- FM3, wrong data: the plan is correct, but the data used is wrong.
- FM4, wrong implementation: the plan is good, but the generated SQL doesn't execute it correctly.
- FM5, runtime error: the generated SQL doesn't even run. Nearly absent from the leaderboard: only four models have any at all, and only one exceeds 2%.
FM2, the wrong plan, dominates by a wide margin: between 53% and 82% of failures depending on the model, for every model with enough failures to be statistically meaningful. In other words, most errors don't come from a model writing bad SQL. They come from a model that never understood which tables to query or how to build its plan before writing a single line of the query itself.
One detail confirms just how independent this is from raw model quality: Fable 5.1, the leaderboard's top model, has the lowest FM2 share in the top ten (57%), but the highest FM3 share (wrong data) on the entire board (28%). Even the best model in the benchmark mostly gets tripped up by what it chooses to read, not by how it expresses that in SQL.
What this means for data engineering
If most failures come from the planning stage, the variable you can actually control isn't which model you pick. It's the quality of what you give it to understand before it plans: the schema, the documentation, the metadata, the metric definitions.
That's exactly the problem I wrote about in my article on the semantic layer: an agent that has to guess what a metric means from ambiguous column names, or choose between three tables that all seem to hold "revenue," is going to plan badly, even with the best model on the market. A documented semantic layer, with a metric dictionary the agent can read directly instead of guessing from the raw schema, cuts down exactly the kind of error FM2 measures: the agent no longer has to infer which tables to use, it's told.
ClickHouse doesn't draw this connection explicitly in the benchmark, but it follows directly from its own conclusion: the bottleneck isn't SQL generation, a task models already handle well (FM5 is nearly nonexistent), it's understanding the business context upstream of that. Improving that understanding is a data-modeling job, not a model-picking one.
What I'd actually do with this benchmark
The natural pull, looking at a leaderboard, is to pick the model on top. The ClickHouse benchmark suggests the opposite: the "best" model depends entirely on your cost constraints, your query volume, and above all the quality of your own data model, which has nothing to do with the one ClickHouse used to build its ranking.
The most useful move, in my view, is to test your own business questions against several models instead of copying the general leaderboard. ClickHouse open-sourced its harness (data-agent-mnist), built to run against any warehouse. Concretely, that means putting together a small set of questions representative of what you actually ask your own agent (not 201, twenty is enough for a first signal), running them through the harness against your own schema, and comparing accuracy, cost, and speed on your own data instead of ClickHouse's synthetic warehouse. The general leaderboard gives you a decent starting intuition, but the only measurement that actually matters is the one done in your own context.
Conclusion
The real variable that decides whether an AI agent answers an analytical question correctly isn't just which model you pick. It's the quality of what you give it to understand before it plans its query. A model at 76.6% that costs 52 times more than one at 65.7% isn't necessarily the right choice: it all depends on what's actually limiting accuracy in your case, the model, or the context you're feeding it.
The full benchmark, its methodology, and the open-source harness are on the ClickHouse blog: clickhouse.com/blog/agentic-analytics-benchmark-data-agent-mnist.