Files

278 lines
7.5 KiB
Markdown
Raw Permalink Normal View History

# Base vs fine-tuned on the same 10 test inputs
First 10 rows of the held-out test split (not cherry-picked). Both models get the identical prompt and identical output post-processing.
### 1. Tell me the ships in classs for kuha class
```sql
-- schema
CREATE TABLE table_name_52 (ships_in_class VARCHAR, class VARCHAR)
```
**Gold**
```sql
SELECT ships_in_class FROM table_name_52 WHERE class = "kuha class"
```
**Base (raw output)**
```
```sql
SELECT DISTINCT ships_in_class FROM table_name_52 WHERE class = 'kuha'
```
```
-> parsed: `SELECT DISTINCT ships_in_class FROM table_name_52 WHERE class = 'kuha'` | exact match: **no**
**Fine-tuned (raw output)**
```
SELECT ships_in_class FROM table_name_52 WHERE class = "kuha"
```
-> parsed: `SELECT ships_in_class FROM table_name_52 WHERE class = "kuha"` | exact match: **no**
---
### 2. What is the Drawn has an Against smaller than 20 and Points of 15?
```sql
-- schema
CREATE TABLE table_name_46 (drawn VARCHAR, against VARCHAR, points VARCHAR)
```
**Gold**
```sql
SELECT drawn FROM table_name_46 WHERE against < 20 AND points = 15
```
**Base (raw output)**
```
```sql
SELECT drawn FROM table_name_46 WHERE against < 20 AND points = '15';
```
```
-> parsed: `SELECT drawn FROM table_name_46 WHERE against < 20 AND points = '15'` | exact match: **no**
**Fine-tuned (raw output)**
```
SELECT drawn FROM table_name_46 WHERE against < 20 AND points = "15"
```
-> parsed: `SELECT drawn FROM table_name_46 WHERE against < 20 AND points = "15"` | exact match: **no**
---
### 3. How many picks for round 12?
```sql
-- schema
CREATE TABLE table_name_5 (pick__number VARCHAR, round VARCHAR)
```
**Gold**
```sql
SELECT COUNT(pick__number) FROM table_name_5 WHERE round = 12
```
**Base (raw output)**
```
SELECT COUNT(pick__number) FROM table_name_5 WHERE round = '12'
```
-> parsed: `SELECT COUNT(pick__number) FROM table_name_5 WHERE round = '12'` | exact match: **no**
**Fine-tuned (raw output)**
```
SELECT COUNT(pick__number) FROM table_name_5 WHERE round = "round 12"
```
-> parsed: `SELECT COUNT(pick__number) FROM table_name_5 WHERE round = "round 12"` | exact match: **no**
---
### 4. wich were the events when bout 2 was zhang ( chn ) l 0-5?
```sql
-- schema
CREATE TABLE table_19398910_4 (event VARCHAR, bout_2 VARCHAR)
```
**Gold**
```sql
SELECT event FROM table_19398910_4 WHERE bout_2 = "Zhang ( CHN ) L 0-5"
```
**Base (raw output)**
```
```sql
SELECT event FROM table_19398910_4 WHERE bout_2 = 'zhang (chn) l 0-5';
```
```
-> parsed: `SELECT event FROM table_19398910_4 WHERE bout_2 = 'zhang (chn) l 0-5'` | exact match: **YES**
**Fine-tuned (raw output)**
```
SELECT event FROM table_19398910_4 WHERE bout_2 = "Zhang ( CHN ) L 0-5"
```
-> parsed: `SELECT event FROM table_19398910_4 WHERE bout_2 = "Zhang ( CHN ) L 0-5"` | exact match: **YES**
---
### 5. Find the name and hours of project that has the most number of scientists.
```sql
-- schema
CREATE TABLE assignedto (project VARCHAR); CREATE TABLE projects (name VARCHAR, hours VARCHAR, code VARCHAR)
```
**Gold**
```sql
SELECT T1.name, T1.hours FROM projects AS T1 JOIN assignedto AS T2 ON T1.code = T2.project GROUP BY T2.project ORDER BY COUNT(*) DESC LIMIT 1
```
**Base (raw output)**
```
```sql
SELECT p.name, p.hours FROM projects AS p JOIN assignedto AS at ON p.code = at.code GROUP BY p.name, p.hours ORDER BY COUNT(*) DESC LIMIT 1;
```
```
-> parsed: `SELECT p.name, p.hours FROM projects AS p JOIN assignedto AS at ON p.code = at.code GROUP BY p.name, p.hours ORDER BY COUNT(*) DESC LIMIT 1` | exact match: **no**
**Fine-tuned (raw output)**
```
SELECT T1.name, T1.hours FROM projects AS T1 JOIN assignedto AS T2 ON T1.code = T2.project GROUP BY T1.name ORDER BY COUNT(*) DESC LIMIT 1
```
-> parsed: `SELECT T1.name, T1.hours FROM projects AS T1 JOIN assignedto AS T2 ON T1.code = T2.project GROUP BY T1.name ORDER BY COUNT(*) DESC LIMIT 1` | exact match: **no**
---
### 6. What is the total number of 3rd placed teams when the host is University of Manitoba, Winnipeg, Manitoba?
```sql
-- schema
CREATE TABLE table_1906920_1 (host VARCHAR)
```
**Gold**
```sql
SELECT COUNT(3 AS rd_place) FROM table_1906920_1 WHERE host = "University of Manitoba, Winnipeg, Manitoba"
```
**Base (raw output)**
```
SELECT COUNT(*) FROM table_1906920_1 WHERE host = 'University of Manitoba' AND position = 3
```
-> parsed: `SELECT COUNT(*) FROM table_1906920_1 WHERE host = 'University of Manitoba' AND position = 3` | exact match: **no**
**Fine-tuned (raw output)**
```
SELECT COUNT(3 AS rd_placed) FROM table_1906920_1 WHERE host = "University of Manitoba, Winnipeg, Manitoba"
```
-> parsed: `SELECT COUNT(3 AS rd_placed) FROM table_1906920_1 WHERE host = "University of Manitoba, Winnipeg, Manitoba"` | exact match: **no**
---
### 7. What is the attendance for week 11?
```sql
-- schema
CREATE TABLE table_name_59 (attendance VARCHAR, week VARCHAR)
```
**Gold**
```sql
SELECT attendance FROM table_name_59 WHERE week = 11
```
**Base (raw output)**
```
SELECT attendance FROM table_name_59 WHERE week = '11'
```
-> parsed: `SELECT attendance FROM table_name_59 WHERE week = '11'` | exact match: **no**
**Fine-tuned (raw output)**
```
SELECT attendance FROM table_name_59 WHERE week = "11"
```
-> parsed: `SELECT attendance FROM table_name_59 WHERE week = "11"` | exact match: **no**
---
### 8. Name the syrian christians for kollam
```sql
-- schema
CREATE TABLE table_23214055_2 (syrian_christians VARCHAR, district VARCHAR)
```
**Gold**
```sql
SELECT syrian_christians FROM table_23214055_2 WHERE district = "Kollam"
```
**Base (raw output)**
```
```sql
SELECT syrian_christians FROM table_23214055_2 WHERE district = 'kollam'
```
```
-> parsed: `SELECT syrian_christians FROM table_23214055_2 WHERE district = 'kollam'` | exact match: **YES**
**Fine-tuned (raw output)**
```
SELECT syrian_christians FROM table_23214055_2 WHERE district = "Kollam"
```
-> parsed: `SELECT syrian_christians FROM table_23214055_2 WHERE district = "Kollam"` | exact match: **YES**
---
### 9. What is every entry for Saturday August 27 when the entry for Thursday August 25 is 23' 56.90 94.528mph?
```sql
-- schema
CREATE TABLE table_30058355_7 (sat_27_aug VARCHAR, thurs_25_aug VARCHAR)
```
**Gold**
```sql
SELECT sat_27_aug FROM table_30058355_7 WHERE thurs_25_aug = "23' 56.90 94.528mph"
```
**Base (raw output)**
```
```sql
SELECT sat_27_aug FROM table_30058355_7 WHERE thurs_25_aug = '23\' 56.90 94.528mph';
```
```
-> parsed: `SELECT sat_27_aug FROM table_30058355_7 WHERE thurs_25_aug = '23\' 56.90 94.528mph'` | exact match: **no**
**Fine-tuned (raw output)**
```
SELECT sat_27_aug FROM table_30058355_7 WHERE thurs_25_aug = "23' 56.90 94.528mph"
```
-> parsed: `SELECT sat_27_aug FROM table_30058355_7 WHERE thurs_25_aug = "23' 56.90 94.528mph"` | exact match: **YES**
---
### 10. Which Owner has a Name / Format of 105.3 kool fm - hot adult contemporary?
```sql
-- schema
CREATE TABLE table_name_18 (owner VARCHAR, name___format VARCHAR)
```
**Gold**
```sql
SELECT owner FROM table_name_18 WHERE name___format = "105.3 kool fm - hot adult contemporary"
```
**Base (raw output)**
```
```sql
SELECT owner FROM table_name_18 WHERE name___format = '105.3 kool fm - hot adult contemporary';
```
```
-> parsed: `SELECT owner FROM table_name_18 WHERE name___format = '105.3 kool fm - hot adult contemporary'` | exact match: **YES**
**Fine-tuned (raw output)**
```
SELECT owner FROM table_name_18 WHERE name___format = "105.3 kool fm - hot adult contemporary"
```
-> parsed: `SELECT owner FROM table_name_18 WHERE name___format = "105.3 kool fm - hot adult contemporary"` | exact match: **YES**
---