boss wants 'chat with your data' by Q4. anyone actually running LLM-to-SQL for business users?
hi all. data analyst kat a bank in KL. my boss went to a vendor event and now he wants "chat with your data" for the business users by Q4 lah. type a question in English or Malay, get the answer from our warehouse. sounds very nice on the slide. my worries, maybe I am overthinking: our schema has 600+ tables, column names like TXN_AMT_2 and CUST_FLG_X . even humans don't know what they mean. "revenue" means 3…
Accepted answer
We ran a pilot of exactly this in Singapore last year. Short answer: it works on a curated subset, never on the raw warehouse. What worked: we built a semantic layer first (about 40 business metrics in dbt, each with one agreed definition, plus descriptions for every column the model is allowed to see). The model writes queries against that layer, not against 600 tables. Accuracy on our test questions went from embarrassing to about 85% on the curated layer. What didn't: anything outside those 40 metrics. And the 15% wrong answers look exactly as confident as the right ones. So the real project is the semantic layer and the definitions. The chatbot is the last 10%.