Enterprise Text-to-SQL

Problem Statement — Spider 90% does not mean enterprise-ready

Spider/BIRD execution accuracy has climbed from 54% to over 90%, but those numbers collapse the moment you move to a real enterprise data lake. LinkedIn set out to build a SQL assistant any employee at the company could use — and this paper is the full engineering write-up of that journey.

1.1 Why academic benchmarks don't carry over

Even Uber's internal QueryGPT only reports roughly 50% overlap on ground-truth tables on their own eval set. In short, slapping a fine-tuned LLM on top of a schema is not enough.

1.2 Four challenges specific to LinkedIn

Three-Layer Architecture

Bottom-up: Knowledge Graph (semantic foundation) → Query Writer Agent (the SQL-generating core agent) → Multi-Agent Chat UI (user interface and intent routing). Data flow between the three layers is shown below.

Each layer in the diagram is expanded in the sections that follow. Three keywords to anchor on: the knowledge graph provides semantics, the Query Writer Agent turns semantics into SQL, and the multi-agent UI handles everything that isn't strictly T2S.

Knowledge Graph — Wiring enterprise semantics into a graph

The central node of the knowledge graph is the Table: from a table you can find its columns, common joins, example queries, domain-knowledge records, and the user groups with read permission. The other organizing node is the Product Area, which ties tables to teams.

3.1 Node attributes (Table 1)

Key design choices: Certification Status flags governance-approved high-quality tables; Column Type distinguishes metrics, dimensions, and attributes so the LLM knows what a column is. When no human description exists, Glean Chat API auto-generates an AI description.

3.2 The five indexes (§2.1.2)

ICA Table Clustering — Picking each team's usual tables out of millions

Millions of tables cannot be fed to an LLM. LinkedIn takes a user-table access matrix from three months of query history and runs Independent Component Analysis (ICA) for soft clustering, mapping each user and product area to its most relevant table subset.

4.1 Pipeline (Algorithm 1)

4.2 Why ICA instead of K-means?

K-means is hard clustering — a table can only go to one cluster, but in reality a member table might be used by growth, ads, and search at once as a "shared table". ICA's soft clustering lets shared tables naturally show up in multiple clusters, and when a team disbands or a table is deprecated, simply rerunning the weekly job re-assigns everything automatically.

4.3 Candidate-table assembly (Algorithm 4)

function GetCandidateTables(user, product_areas):
    user_clusters        = GetUserGroupClusters(user)
    product_email_groups = GetEmailGroups(product_areas)
    representative_users = GetEmployees(product_email_groups)
    product_area_clusters = GetUserGroupClusters(representative_users)
    clusters             = merge(user_clusters, product_area_clusters)

    inferred_tables = GetExtendedTables(clusters)
    explicit_tables = GetExplicitTables(product_areas)

    return merge(inferred_tables, explicit_tables)

At the start of every chat session, this function flattens all clusters corresponding to the user + selected product areas and adds in manually-curated explicit tables — defining the search scope for the subsequent EBR. Long-tail tables outside this set are dynamically pulled in via in-chat follow-up.

Query Writer Agent — 4 steps to generate SQL

Implemented in langchain. Inputs: question, user name, list of product areas. Output: a dict containing the SQL, an explanation, the tables/columns used, and assumptions for the user to verify.

5.1 Step 1 — Retrieve Context

Goal: high recall. First the candidate-table list from §04 bounds the search space, then we pull from three sources:

1. EBR with user question — query the table index with the question's embedding; results restricted to candidate tables.
2. Tables in retrieved examples — first retrieve example queries similar to the user's question, then extract the tables those examples use.
3. Tables in user question — pick up any table names the user explicitly mentioned — this path is NOT restricted to the candidate set; if the user wants it, use it.

The three combined produce K_ret = 20 tables, then we fetch all their columns, plus the product area's domain knowledge and jargon entries.

5.2 Step 2 — Rank Context (Table Ranker + Column Ranker)

An LLM trims 20 tables down to K_rnk = 7. The Table Ranker LLM scores each table 1–10 with explanations, using:

• • table name and description
• • example queries that use the table
• • commonly joined / co-queried tables
• • domain knowledge and jargon explanations

A notable callout from the authors: feeding the table schema to the ranker actually hurt recall and increased latency, so schemas are omitted here.

The Column Ranker then selects columns from the top 7 tables into two tiers: relevant and potentially relevant. Including the second tier improves recall. Each table is rendered as a CREATE TABLE statement with columns ordered by usage popularity.

5.3 Step 3 — Write Query

The selected tables + columns + relevance scores and explanations + examples + domain knowledge + jargon all get packed into the prompt. The Query Writer LLM (gpt-4o) returns:

{
  "assumptions": [...],   # for the user to verify
  "query": "SELECT ...",
  "explanation": "...",
  "tables": [...],
  "columns": [...]
}

5.4 Step 4 — Validate & Fix (Researcher Agent enters)

The query validation loop runs at most twice, handling two error categories:

5.5 Researcher LLM Agent

The Researcher is a self-reflecting agent built specifically for hallucination resolution. It has tools to search tables, fetch table schemas, and fetch metadata. For instance, it can search for the table most similar to a hallucinated one.

5.6 Hyperparameters at a glance

Multi-Agent UI — The interface may matter more than the LLM

The biggest lesson the authors learned post-launch: users don't want Text-to-SQL — they want a general data assistant that can write queries, find tables, debug bugs, and answer any data question. That's why the system evolved into a multi-agent structure.

6.1 Intent classifier routes to four agents

6.2 Speed tricks for the Q&A agent

The Q&A agent has the widest scope, so the authors use three speed-ups: ① evaluate question difficulty and skip self-reflection on simple ones; ② pre-fetch table metadata to avoid repeated tool calls; ③ both LLMs use gpt-4o-mini.

6.3 UI design highlights

Evaluation — Metrics and Ablations

The authors built an internal benchmark: 133 questions × 10 product areas × 167 ground-truth tables, with 60% of questions allowing multiple correct answers (multiple ground truths).

7.1 Three metric categories

7.2 Scoring rubric (Figure 4)

7.3 Ablation study (Table 2) — full reproduction

Models: example embeddings use E5-large-v2, table/column embeddings use text-embedding-ada-002, Researcher LLM uses gpt-4o-mini-2024-07-18, other LLMs use gpt-4o-2024-05-13, temperature = 0, single run.

7.4 Key observations

Production Numbers

The chatbot has been live since July 2024. Below are the real usage numbers at the time of writing (April 2025).

8.1 Adoption

8.2 Expert review

Domain experts from each of the 10 product areas rated the production version, returning ratings for 124 of 133 questions:

8.3 User satisfaction survey

Negative Results — Academic tricks that didn't work here

This section is the most valuable part of the paper. The authors tried three mainstream academic techniques, and all three failed in the LinkedIn setting. Free lessons for anyone building enterprise T2S.

9.1 Multi-query + Self-Consistency: dies without query execution

The academic recipe is to generate multiple queries and pick the most self-consistent (e.g. CHASE-SQL, CHESS). But in an enterprise setting:

• ❌ No data access — can't execute queries to compare outputs
• ❌ Users can't wait — multi-round generation noticeably increases latency

A workaround (generating up to 3 ways to write the query as a kind of CoT) did not improve recall. The authors' conclusion: self-consistency's effectiveness comes from input shuffling AND actual execution — neither can be skipped.

9.2 Query Planner LLM Decomposition: smaller subtasks make things worse

Following DIN-SQL — use a planner LLM to break the question into subtasks solved iteratively. Result:

• ❌ Adds extra latency
• ❌ Generated queries become overly nested
• ❌ Both recall and quality drop

Even when they constrained the planner to "minimize the number of tasks" with few-shot examples, it still failed. The hypothesis: gpt-4o is strong enough on its own that task decomposition only constrains its solution space — smaller models may need it, but gpt-4o doesn't.

9.3 Using planner subtasks for query expansion: no EBR gain

Use the planner's subtasks as query strings for retrieval (a form of query expansion). The result: no improvement in EBR table recall. The authors speculate that a prompt explicitly designed for query expansion might work, but the subtask format isn't quite right.

Conclusion & Takeaways

The authors claim this is the first publicly available, fully detailed enterprise Text-to-SQL paper. Required reading for any team building an internal SQL chatbot.

10.1 Quantitative summary

10.2 Caveats the paper doesn't dwell on but are worth noting