135 lines
5.2 KiB
Markdown
135 lines
5.2 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: Qwen/Qwen2.5-3B-Instruct
|
||
tags:
|
||
- text-to-sql
|
||
- nl2sql
|
||
- spider
|
||
- grpo
|
||
- reinforcement-learning
|
||
- tool-use
|
||
- verl
|
||
- oumi
|
||
language:
|
||
- en
|
||
datasets:
|
||
- spider
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# Qwen2.5-3B-Instruct — NL2SQL Spider GRPO (step 100)
|
||
|
||
`Qwen/Qwen2.5-3B-Instruct` trained with **GRPO** in a tool-calling environment where the
|
||
model issues real `run_sql` queries against live SQLite databases and is rewarded on whether
|
||
its final query returns the same rows as the gold query.
|
||
|
||
Trained with [verl](https://github.com/volcengine/verl) 0.7.1 through
|
||
[Oumi](https://github.com/oumi-ai/oumi)'s `DatabaseExecutableEnvironment`.
|
||
|
||
## Results — Spider dev (1034 rows, greedy, single sample)
|
||
|
||
| metric | base | **this model** | change |
|
||
|---|---|---|---|
|
||
| **official test-suite execution accuracy** | 0.336 | **0.644** | **×1.92** |
|
||
| official exact-set-match | 0.195 | 0.397 | ×2.04 |
|
||
| legacy `evaluation.py` execution | 0.264 | 0.583 | ×2.21 |
|
||
|
||
**Quote the test-suite number (0.644).** It is the current official Spider metric
|
||
(Zhong, Yu & Klein, EMNLP 2020, [`taoyds/test-suite-sql-eval`](https://github.com/taoyds/test-suite-sql-eval)).
|
||
The legacy `evaluation.py` under-reports this model by 6.1 points because 242 of its 1034
|
||
predictions cannot be parsed by Spider's `process_sql.py` (subqueries, trailing `;`, `AS`
|
||
aliases), and an unparseable prediction scores 0 regardless of the rows it returns.
|
||
|
||
### By difficulty (test-suite)
|
||
|
||
| | easy (248) | medium (446) | hard (174) | extra (166) | all |
|
||
|---|---|---|---|---|---|
|
||
| this model | 0.863 | 0.668 | 0.592 | 0.307 | **0.644** |
|
||
| base | 0.431 | 0.370 | 0.322 | 0.114 | 0.336 |
|
||
|
||
Multi-table reasoning is the bottleneck: easy 0.86 → extra-hard 0.31.
|
||
|
||
## Intended use
|
||
|
||
A research artifact demonstrating that **execution feedback works as an RL signal** for
|
||
text-to-SQL. It is not a state-of-the-art Spider model — published 3B systems reach 79–84%
|
||
using constrained decoding, schema ranking, full SFT, and multi-candidate inference, none of
|
||
which are used here. The result is 100 RL steps on a generic (non-code) instruct model.
|
||
|
||
## How it was trained
|
||
|
||
| | |
|
||
|---|---|
|
||
| algorithm | GRPO (`adv_estimator: grpo`, `norm_adv_by_std_in_grpo: true`), KL loss disabled |
|
||
| hardware | 8×H100, single node, ~80 s/step, ~2h30m total |
|
||
| batch | 64 prompts × n=8 = 512 trajectories/step |
|
||
| completion budget | 2048 tokens shared across all turns |
|
||
| lr | 1e-6 constant, AdamW |
|
||
| sampling | train temp 1.0; eval greedy (t=0, n=1) |
|
||
| reward | binary result-set match against the gold query |
|
||
| train data | 6,997 of 7,000 `train_spider.json` rows (3 dropped — their *gold* SQL does not execute) |
|
||
| eval data | 1,034 `dev.json` rows, none dropped |
|
||
|
||
The reward is outcome-only: it looks at the final SQL and nothing else — not turn count, not
|
||
whether a tool was called, not style.
|
||
|
||
## Prompt format
|
||
|
||
The model expects a system message containing the full DDL of every table in the target
|
||
database, a user question, and hermes-style tool calling with a single `run_sql(query)` tool.
|
||
It answers with a fenced ```sql block.
|
||
|
||
```
|
||
[system] You are a SQL assistant for a SQLite database with this schema:
|
||
CREATE TABLE singer (...)
|
||
You may call the tool run_sql(query) to execute a SQL query and inspect
|
||
results. When you are done, give your final answer as a single SQL query
|
||
inside a ```sql code block.
|
||
[user] How many singers do we have?
|
||
```
|
||
|
||
```
|
||
[assistant] <tool_call>{"name":"run_sql","arguments":{"query":"SELECT COUNT(*) FROM singer;"}}</tool_call>
|
||
[tool] {"columns": ["COUNT(*)"], "rows": [[6]]}
|
||
[assistant] ```sql
|
||
SELECT COUNT(*) FROM singer;
|
||
```
|
||
```
|
||
|
||
**To reproduce the numbers above you must replay the training tool loop.** One-shot prompting
|
||
measures a different system and understates badly. Tool responses must be truncated to 256
|
||
characters, middle-elided, exactly as verl does during training:
|
||
|
||
```python
|
||
half = max_length // 2
|
||
text = text[:half] + "...(truncated)..." + text[-half:]
|
||
```
|
||
|
||
## Behavioral profile
|
||
|
||
| | base | this model |
|
||
|---|---|---|
|
||
| valid executable SQL | — | 94% |
|
||
| answers with a query it actually ran | 43.9% | **85.4%** |
|
||
| `run_sql` calls per row | 1.79 | 1.02 |
|
||
| predictions unparseable by Spider's parser | 62.6% | 24.0% |
|
||
|
||
Training made tool use *precise* rather than eliminating it, and made SQL markedly more
|
||
standard despite the reward never mentioning style. The dominant remaining failure is **197
|
||
rows (19.1%) that run a query, see real rows, and still answer wrong** — schema grounding on
|
||
multi-table questions, not tool use and not syntax.
|
||
|
||
## Limitations
|
||
|
||
- Single seed, no error bars, one run per configuration.
|
||
- Evaluated on Spider **dev**. The held-out test split was never touched.
|
||
- Greedy single-sample only; no self-consistency or multi-candidate reranking.
|
||
- A later checkpoint (step 250) scores higher on the internal metric (0.752 vs 0.727) but was
|
||
never run through official eval. **This checkpoint (step 100) is the one every number above
|
||
refers to.**
|
||
|
||
## Citation
|
||
|
||
Environment and reward implementation: [oumi-ai/oumi](https://github.com/oumi-ai/oumi)
|
||
PRs #2560 (verl tool adapter), #2561 (NL2SQL env + reward), #2562 (Spider demo).
|