Skip to content
question

boss wants 'chat with your data' by Q4. anyone actually running LLM-to-SQL for business users?

Nurul Aisyah@nurul_sql

August 22, 2026 · 1 min read

#2 in My Model Is Fine, The Data Is Not

hi all. data analyst kat a bank in KL. my boss went to a vendor event and now he wants "chat with your data" for the business users by Q4 lah. type a question in English or Malay, get the answer from our warehouse. sounds very nice on the slide.

my worries, maybe I am overthinking:

  • our schema has 600+ tables, column names like TXN_AMT_2 and CUST_FLG_X. even humans don't know what they mean.
  • "revenue" means 3 different things depending which department asks. finance number and dashboard number already don't match, now got a third one from the bot?
  • if the SQL is wrong but looks right, the business user won't know. they will put it in a board deck. alamak.
  • bank data. what can go to the model, what cannot.

anyone running this in production for real? what worked, what didn't? can or not, honestly?

Accepted answer

Lin Wei#1 in My Model Is Fine, The Data Is Not4 likes

We ran a pilot of exactly this in Singapore last year. Short answer: it works on a curated subset, never on the raw warehouse. What worked: we built a semantic layer first (about 40 business metrics in dbt, each with one agreed definition, plus descriptions for every column the model is allowed to see). The model writes queries against that layer, not against 600 tables. Accuracy on our test questions went from embarrassing to about 85% on the curated layer. What didn't: anything outside those 40 metrics. And the 15% wrong answers look exactly as confident as the right ones. So the real project is the semantic layer and the definitions. The chatbot is the last 10%.

View in the thread

#sql#llm#data-science#databases

Get the best of #my-model-is-fine weekly

One email each Monday with the most useful posts. No spam; unsubscribe in one click.

Comments

8

Sign in to join the conversation.

Sign in to comment

Accepted answer

Lin Wei@linwei#1 in My Model Is Fine, The Data Is Not
We ran a pilot of exactly this in Singapore last year. Short answer: it works on a curated subset, never on the raw warehouse. What worked: we built a semantic layer first (about 40 business metrics in dbt, each with one agreed definition, plus descriptions for every column the model is allowed to see). The model writes queries against that layer, not against 600 tables. Accuracy on our test questions went from embarrassing to about 85% on the curated layer. What didn't: anything outside those 40 metrics. And the 15% wrong answers look exactly as confident as the right ones. So the real project is the semantic layer and the definitions. The chatbot is the last 10%.
4 net votes
Nurul Aisyah@nurul_sql#2 in My Model Is Fine, The Data Is Not
so the bot is the easy part, the boring metric definitions is the actual project. confirm my boss won't like this answer but at least I can show him someone really did it 🙏 thanks!
0 net votes
Mo Siddiqui@mo_boston#4 in My Model Is Fine, The Data Is Not
+1 to lin. also show the generated sql to the user every time, even if they can't read it. our analysts can, and they caught a join that doubled revenue in week 1 lol
3 net votes
Dana Kowalski@dana_ml#3 in My Model Is Fine, The Data Is Not
Two things from the evaluation side. First, before the pilot, collect 100 real questions from business users with the correct answer written by your team. That is your eval set, and it makes the board-deck risk measurable: how often is the answer wrong while looking plausible? Second, allow the system to say 'I do not know which revenue you mean' and score that as a success. A system that asks a clarifying question is better than one that guesses, and most demos punish asking.
2 net votes
Nurul Aisyah@nurul_sql#2 in My Model Is Fine, The Data Is Not
the 100 real questions idea is very good. I already have like 300 'can you just pull the numbers' emails from Friday 5pm, that's my eval set 😂
1 net votes
Dana Kowalski@dana_ml#3 in My Model Is Fine, The Data Is Not
That is the best eval set I have heard of. Real questions, real ambiguity, real deadlines.
0 net votes
Danny Cho@danny_cho_la#10 in My Model Is Fine, The Data Is Not
Also: give the bot read access to a replica only, with row limits and a query timeout. Someone will ask 'show me all transactions' and you do not want that hitting your primary at month end.
2 net votes
Zhang Xinyi 张欣怡@xinyi_in_melb#5 in My Model Is Fine, The Data Is Not
Hi Nurul, I found this thread from your profile. For my capstone we did a small text-to-SQL demo and my supervisor said exactly the same thing as lin, the schema descriptions matter more than the model. Very useful to see the real bank version, thanks!! 🙏
0 net votes