47 lines
1.5 KiB
Markdown
47 lines
1.5 KiB
Markdown
|
---
|
|||
|
|
library_name: transformers
|
|||
|
|
license: other
|
|||
|
|
base_model: Qwen/Qwen3-1.7B
|
|||
|
|
tags:
|
|||
|
|
- llama-factory
|
|||
|
|
- full
|
|||
|
|
- generated_from_trainer
|
|||
|
|
- routing
|
|||
|
|
- classifier
|
|||
|
|
model-index:
|
|||
|
|
- name: societas-router-lenv3-17b
|
|||
|
|
results: []
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# societas-router-lenv3-17b
|
|||
|
|
|
|||
|
|
A per-call **LLM routing classifier** — a full-parameter fine-tune of
|
|||
|
|
[Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B). At each step of an agent's trajectory it
|
|||
|
|
reads a read-only snapshot of the trajectory-so-far and predicts the capability **tier** the *next*
|
|||
|
|
model call needs, emitting a single JSON verdict `{"tier": "...", "reason": "..."}` with
|
|||
|
|
`tier ∈ {Routine, Medium, Advanced}`.
|
|||
|
|
|
|||
|
|
## Results (held-out test split, 251 examples)
|
|||
|
|
|
|||
|
|
- **Accuracy: 60.16%** (151/251), `parse_fail = 0`, no think-token leakage. (chance = 33%)
|
|||
|
|
- Per-tier recall: Routine 58.3% · Medium 52.4% · Advanced 69.9%.
|
|||
|
|
|
|||
|
|
## Usage note (important)
|
|||
|
|
|
|||
|
|
Trained with the Qwen3 chat template at `enable_thinking=False` (the empty `<think></think>` block is
|
|||
|
|
in the prompt; the JSON verdict is the target). **Serve it the same way** — render prompts with
|
|||
|
|
`enable_thinking=False`.
|
|||
|
|
|
|||
|
|
## Training procedure
|
|||
|
|
|
|||
|
|
Full-parameter SFT via LLaMA-Factory, bf16, on 2× A800-80G.
|
|||
|
|
|
|||
|
|
- learning_rate: 2e-05, cosine schedule, warmup_ratio 0.03
|
|||
|
|
- per_device_train_batch_size: 1, gradient_accumulation_steps: 8, total_train_batch_size: 16
|
|||
|
|
- num_epochs: 3.0, cutoff_len (max context): 16384, packing: off
|
|||
|
|
- optimizer: AdamW (torch), betas=(0.9,0.999), eps=1e-08
|
|||
|
|
|
|||
|
|
### Framework versions
|
|||
|
|
|
|||
|
|
- Transformers 4.53.x / Pytorch 2.5.1+cu121 / Datasets / Tokenizers
|