70 lines
2.8 KiB
Markdown
70 lines
2.8 KiB
Markdown
---
|
||
base_model: Qwen/Qwen3-1.7B
|
||
license: apache-2.0
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
tags:
|
||
- qwen3
|
||
- cogs
|
||
- json
|
||
- structured-output
|
||
- rag
|
||
- mlx
|
||
---
|
||
|
||
# qwen3-1.7b-cogs-ask
|
||
|
||
**Merged** full model (Qwen/Qwen3-1.7B + LoRA, fused, bf16) — the student model
|
||
for the [cogs](https://github.com/trunksio/cogs) `ask` pipeline. Standalone
|
||
adapter: [`lewisdog/qwen3-1.7b-cogs-ask-lora`](https://huggingface.co/lewisdog/qwen3-1.7b-cogs-ask-lora).
|
||
Sibling model for the ingest pipeline:
|
||
[`lewisdog/qwen3-1.7b-cogs-ingest`](https://huggingface.co/lewisdog/qwen3-1.7b-cogs-ingest).
|
||
|
||
Two tasks, both emitting compact JSON exactly as the cogs runtime parses it:
|
||
|
||
| task | output JSON |
|
||
|--------------|-------------------------------------------------------------------|
|
||
| `decompose` | `{"subquestions": [...]}` — question → 1-4 sub-questions |
|
||
| `synthesize` | `{"answer": "... [note-id] ...", "citations": [...], "abstained": bool}` — grounded answer over retrieved notes; **abstains** on out-of-domain questions |
|
||
|
||
## Eval (full 63-record valid set, temp 0, repetition_penalty 1.1)
|
||
|
||
- JSON parse: **100%** · decompose non-empty: **100%**
|
||
- synthesize citation-validity: **77%** (→ **~90%** with the prefix-repair below)
|
||
- abstention preserved: **100%** · false-abstention: **4%**
|
||
|
||
## Apple silicon (MLX / omlx)
|
||
|
||
```sh
|
||
python3 -m mlx_lm.convert --hf-path lewisdog/qwen3-1.7b-cogs-ask -q --q-bits 8 --mlx-path qwen3-cogs-ask-mlx
|
||
# NOTE: if transformers writes v5-format configs, backfill rope_theta / rope_scaling /
|
||
# torch_dtype from the base Qwen3-1.7B config and swap in the original Qwen3-1.7B
|
||
# tokenizer files before converting. Wire as [llm.ask] in cogs.toml.
|
||
```
|
||
|
||
## ⚠️ Serving notes
|
||
|
||
1. **Decode at temp 0 + `repetition_penalty=1.1`** (pure greedy runs away on long
|
||
list outputs), `enable_thinking=False`, stop on `<|im_end|>`.
|
||
2. **Repair citation prefixes.** Most citation errors are a dropped/wrong
|
||
namespace prefix (`enisa-…` → `sources-enisa-…`). Normalize against the notes
|
||
actually in the prompt and drop anything unverifiable:
|
||
|
||
```python
|
||
PFX = ("sources-", "source-", "concepts-", "concept-", "entities-", "entity-")
|
||
core = lambda x: next((x[len(p):] for p in PFX if x.startswith(p)), x)
|
||
def repair(cite, note_ids):
|
||
if cite in note_ids: return cite
|
||
return {core(n): n for n in note_ids}.get(core(cite)) # None => drop it
|
||
```
|
||
3. Abstention is reliable — trust it to decline out-of-domain questions.
|
||
|
||
## Training
|
||
|
||
LoRA r=32/α=64, 5 epochs, eff. batch 16, max_seq 8192, lr 1e-4 cosine, bf16, on a
|
||
DGX Spark (GB10). Data: `cogs distill` ask pairs (475 base + abstentions
|
||
oversampled 3×). Eval loss 1.906 → 0.758; eval token-acc 0.836. Full write-up in
|
||
the repo `RESULTS.md`.
|
||
|
||
- PEFT 0.19.1 · TRL 1.7.1 · Transformers 5.13.0 · PyTorch 2.12.1+cu130
|