83 lines
3.1 KiB
Markdown
83 lines
3.1 KiB
Markdown
---
|
|
base_model: Qwen/Qwen3-1.7B
|
|
license: apache-2.0
|
|
pipeline_tag: text-generation
|
|
library_name: transformers
|
|
tags:
|
|
- qwen3
|
|
- cogs
|
|
- json
|
|
- structured-output
|
|
- mlx
|
|
---
|
|
|
|
# qwen3-1.7b-cogs-ingest
|
|
|
|
**Merged** full model (Qwen/Qwen3-1.7B + LoRA, weights fused, bf16) — the student
|
|
model for the [cogs](https://github.com/trunksio/cogs) `ingest` pipeline. Use
|
|
this repo when you want a standalone model to serve or convert; the standalone
|
|
LoRA adapter is at
|
|
[`lewisdog/qwen3-1.7b-cogs-ingest-lora`](https://huggingface.co/lewisdog/qwen3-1.7b-cogs-ingest-lora).
|
|
|
|
It replaces a large teacher LLM as a local, OpenAI-compatible provider for four
|
|
structured-output tasks, emitting **compact JSON exactly as the cogs runtime
|
|
parses it**:
|
|
|
|
| task | required top-level JSON keys |
|
|
|-----------------|---------------------------------------------|
|
|
| `extract` | `summary`, `key_claims` (+ quotes/entities) |
|
|
| `suggest_links` | `linked_claims` |
|
|
| `page_update` | `topic`, `section_md`, `relevant` |
|
|
| `contradiction` | `findings` |
|
|
|
|
## Apple silicon (MLX / omlx)
|
|
|
|
```sh
|
|
# converts + 4-bit quantizes into an MLX model dir
|
|
python3 -m mlx_lm.convert --hf-path lewisdog/qwen3-1.7b-cogs-ingest -q --mlx-path qwen3-cogs-ingest-mlx
|
|
# then drop into ~/.omlx/models and point cogs [llm] at it
|
|
```
|
|
|
|
## ⚠️ Serving: avoid pure greedy decoding
|
|
|
|
The model learned the schema well, but under **pure greedy (temperature 0, no
|
|
penalty)** it degenerates on the long list-valued tasks (`extract`,
|
|
`suggest_links`): it keeps appending list items and never emits `<|im_end|>`,
|
|
leaving valid-but-**unterminated** JSON. Fix with either:
|
|
|
|
- `repetition_penalty ≈ 1.1` (keeps determinism), **or**
|
|
- Qwen non-thinking sampling: `temperature=0.7, top_p=0.8, top_k=20`
|
|
(this model's bundled `generation_config.json` already samples at temp 0.6,
|
|
which sidesteps the issue — just don't override it to temp-0 greedy).
|
|
|
|
With either, a 5-sample / 4-task JSON parse eval is **5/5 (100%) schema-exact**
|
|
(vs. 3/5 at plain greedy). Use `enable_thinking=False` and stop on `<|im_end|>`.
|
|
|
|
## Transformers usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
"lewisdog/qwen3-1.7b-cogs-ingest", dtype="bfloat16", device_map="auto"
|
|
).eval()
|
|
tok = AutoTokenizer.from_pretrained("lewisdog/qwen3-1.7b-cogs-ingest")
|
|
|
|
enc = tok.apply_chat_template(
|
|
messages, add_generation_prompt=True, enable_thinking=False,
|
|
return_tensors="pt", return_dict=True,
|
|
).to(model.device)
|
|
out = model.generate(**enc, max_new_tokens=2048, do_sample=False,
|
|
repetition_penalty=1.1, pad_token_id=tok.pad_token_id)
|
|
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
|
|
```
|
|
|
|
## Training
|
|
|
|
LoRA SFT (TRL) on 1,921 `cogs distill` chat pairs, 2 epochs, effective batch 16,
|
|
max_seq 8192, lr 1e-4 cosine, bf16, on an NVIDIA DGX Spark (GB10).
|
|
Train loss 2.55 → 1.41; eval loss 1.125 → 1.118 (still decreasing); eval
|
|
token-accuracy 0.756. Full details: adapter repo card + `RESULTS.md`.
|
|
|
|
- PEFT 0.19.1 · TRL 1.7.1 · Transformers 5.13.0 · PyTorch 2.12.1+cu130
|