An LLM Can Write the SQL. It Still Doesn't Know What Revenue Means.

Ask an LLM for last quarter's revenue by channel. Four seconds later you have clean SQL against your own warehouse.

Small problem. Your warehouse has six versions of revenue.

One nets out refunds and one doesn't. One books at contract signature, another at fulfillment. One is a stale view nobody deprecated, because deleting a revenue table is a very specific kind of terrifying. And one was built for a 2023 board deck and never touched again.

The model picks one. Confidently. Nobody wrote down which one the CFO means, so it guesses. When it misses, it does so quietly. Quietly wrong is worse than slowly wrong.

This is where the demo usually ends.

The query-writing problem got smaller. Fast.

If your objection to text-to-SQL is that models can't handle a real warehouse, that objection is expiring. In November 2024 a research group published Spider 2.0, a benchmark of 632 real enterprise text-to-SQL problems. Real schemas. Thousand-column tables. Cloud projects with dialect quirks and their own codebases. Roughly the mess you have in production.

Spider 2.0 made the gap between demos and production obvious. The original code-agent baseline solved 21.3% of the full benchmark, compared with 91.2% on Spider 1.0.

Since then, progress has been fast. On the current Spider2-Snow leaderboard, the top submissions score above 90. That is not a perfect apples-to-apples comparison. The task setting changed. Still, the direction is hard to miss.

So the machines are getting very good at writing queries. I use these tools. They're good.

The query is finished. The argument is just starting.

The benchmark has an answer key. You don't.

Every score on that leaderboard rests on one quiet detail: each task has a gold answer. A human read the schema, decided what the question meant, wrote the correct query, and recorded the right result. The agent gets graded against a definition somebody already agreed on.

Your company skipped that step.

There is no gold answer for "revenue" at your company. There are four defensible answers and two indefensible ones, and the person who can tell them apart is in a meeting until Thursday.

A wrong number that took a week had a person attached to it. You could ask what they included. They'd wince and say the view double-counts renewals. A wrong number that took four seconds has no history, no owner, no wince, and perfect formatting. It looks exactly like a right number.

The bottleneck was never SQL syntax. It was agreement.

Why nobody writes the definitions down

The fix is not mysterious. Write the definitions down. Call it a semantic layer if a vendor is in the room. Metric definitions that live in code, versioned, owned, one place where "revenue" resolves to one expression. The tooling for this already exists, and most of it is fine. It doesn't even have to be tooling. A page that says what revenue means, signed off by finance, probably puts you ahead of a lot of companies with seven-figure data stacks.

Almost nobody does it. My read: incentives, not ignorance.

Writing the definition down forces the fight. The moment "revenue" has to resolve to one expression, finance and sales discover they've been reporting different things for two years. One of those teams has been reporting the more flattering number. Somebody's dashboard gets corrected downward, in public.

Nobody wants that meeting.

Skipping it costs nothing this quarter, to any individual person. The company pays later, collectively. That's the kind of bill that never lands on one desk, which is exactly why it keeps not getting paid.

Meanwhile there's a senior analyst somewhere who knows which table the CFO means, from two years of getting corrected in meetings. That person is the semantic layer. Not in code. In Slack. Also, being the only one who can reconcile the numbers is not exactly bad job security.

Most companies would rather buy a tool than schedule an argument. The tool has a demo. The argument doesn't.

Cheap queries, expensive meetings

Is this a real cost or a data-team pet complaint? Fair question. Say an analyst spends three days a month reconciling totals that disagree. That's the low end of what I've watched happen inside real companies. A seventh of an expert's time, every month, spent producing not insight but agreement.

Now hand every department a text-to-SQL tool with no shared definitions under it. The reconciliation work didn't go away. It got more inputs. The marketing lead and the ops lead can each generate a confident revenue number before lunch, from different tables, and both of them walk into Monday's meeting armed. Same warehouse. Same tool. Two confident numbers. Zero agreement. Nobody makes a decision. Everyone leaves with homework.

Cheap querying without agreed definitions is a machine for manufacturing disputes at scale.

The boring advantage

My bet: written definitions become the sorting mechanism over the next few years. Companies that have them plug these tools in and get a real multiplier. Questions answered in seconds, against numbers everyone already signed off on. Companies that don't get faster wrong answers, in volume, in a confident tone.

Timing is the part I'm least sure of. The capability curve says soon. The org chart says nothing moves until a wrong number costs someone a bad quarter in public.

The models will keep getting better at writing your queries. Nobody is coming to write down what your numbers mean.