20 lines
678 B
Markdown
20 lines
678 B
Markdown
|
# External ToolACE Evaluation
|
||
|
|
|
||
|
|
- Base: `openbmb/MiniCPM5-1B`
|
||
|
|
- Adapter: `./minicpm5_tooluse_nemotron_dpo_from_repair_dpo`
|
||
|
|
- Eval dataset: `Team-ACE/ToolACE`
|
||
|
|
- Cases: `300`
|
||
|
|
|
||
|
|
| Metric | Base | Final adapter | Delta |
|
||
|
|
|---|---:|---:|---:|
|
||
|
|
| parseable_rate | 0.0133 | 0.9933 | +0.9800 |
|
||
|
|
| valid_name_rate | 0.0133 | 0.9700 | +0.9567 |
|
||
|
|
| expected_name_rate | 0.0133 | 0.9267 | +0.9133 |
|
||
|
|
| args_exact_rate | 0.1500 | 0.6533 | +0.5033 |
|
||
|
|
| arg_key_overlap | 0.0033 | 0.7517 | +0.7484 |
|
||
|
|
| no_schema_copy_rate | 1.0000 | 1.0000 | +0.0000 |
|
||
|
|
| no_repetition_rate | 0.9967 | 1.0000 | +0.0033 |
|
||
|
|
| stopped_cleanly_rate | 0.0000 | 0.1500 | +0.1500 |
|
||
|
|
|
||
|
|
Final adapter beats base on `7/8` metrics.
|