98 lines
3.4 KiB
Markdown
98 lines
3.4 KiB
Markdown
---
|
|
license: mit
|
|
base_model:
|
|
- WeiboAI/VibeThinker-3B
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- unsloth
|
|
- lora
|
|
- trl
|
|
- sft
|
|
---
|
|
|
|
# VibeThinker-3B-SFT-Fable5-Glint
|
|
|
|
A LoRA fine-tune of [`WeiboAI/VibeThinker-3B`](https://huggingface.co/WeiboAI/VibeThinker-3B), supervised fine-tuned on `ermiaazarkhalili/Fable-5-Glint-Clean` (private).
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| **Base model** | [`WeiboAI/VibeThinker-3B`](https://huggingface.co/WeiboAI/VibeThinker-3B) |
|
|
| **Architecture** | `Qwen2ForCausalLM` |
|
|
| **Parameters** | 3.1B |
|
|
| **Training data** | `ermiaazarkhalili/Fable-5-Glint-Clean` (private) |
|
|
| **Method** | LoRA supervised fine-tuning via [Unsloth](https://github.com/unslothai/unsloth) + [TRL](https://github.com/huggingface/trl) |
|
|
| **License** | `mit` (inherited from the base model) |
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "ermiaazarkhalili/VibeThinker-3B-SFT-Fable5-Glint"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, dtype='auto', device_map='auto')
|
|
|
|
messages = [{"role": "user", "content": "Explain gradient checkpointing in two sentences."}]
|
|
inputs = tokenizer.apply_chat_template(
|
|
messages, add_generation_prompt=True, return_tensors='pt'
|
|
).to(model.device)
|
|
|
|
outputs = model.generate(inputs, max_new_tokens=256)
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
## Training configuration
|
|
|
|
| Setting | Value |
|
|
| --- | --- |
|
|
| LoRA rank (r) | 16 |
|
|
| LoRA alpha | 16 |
|
|
| Learning rate | 0.0002 |
|
|
| Epochs | 3 |
|
|
| Effective batch size | 8 (2 x 4 grad accum) |
|
|
| Max sequence length | 4096 |
|
|
| Base precision | 4-bit (QLoRA) |
|
|
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
|
|
|
|
## Held-out evaluation
|
|
|
|
Next-token accuracy on a deterministic held-out split of `ermiaazarkhalili/Fable-5-Glint-Clean` (n = 199 samples), scored against the base model [`WeiboAI/VibeThinker-3B`](https://huggingface.co/WeiboAI/VibeThinker-3B). Assistant tokens only; both models are scored identically.
|
|
|
|
| Metric | Base | This model | Δ |
|
|
| --- | ---: | ---: | ---: |
|
|
| Top-1 accuracy | 0.5309 | 0.6719 | +0.1411 |
|
|
| Top-5 accuracy | 0.7563 | 0.8947 | +0.1384 |
|
|
|
|
A delta measures how far fine-tuning moved this model from its own starting point; it is not a ranking against other models, which start from different baselines.
|
|
|
|
## Observed training loss
|
|
|
|
Measured from our SLURM logs for this configuration. These are training-loss
|
|
observations only — see the held-out evaluation above for measured accuracy.
|
|
|
|
| SLURM job | Steps | First loss | Final loss |
|
|
| --- | --- | --- | --- |
|
|
| `45987994` | 1,554 | 1.7458 | 1.0123 |
|
|
| `46021015` | 1,554 | 1.7457 | 1.0127 |
|
|
|
|
## Limitations
|
|
|
|
- No benchmark evaluation has been run on this checkpoint. The only reported
|
|
numbers are training-loss observations.
|
|
- Inherits the biases, knowledge cutoff and failure modes of the base model.
|
|
- Fine-tuned on a single instruction-following dataset; behaviour outside that
|
|
distribution is untested.
|
|
- LoRA adapters were merged into the base weights, so the merged model cannot
|
|
be detached from this fine-tune.
|
|
|
|
## Reproducing
|
|
|
|
Trained by `notebooks/fable_distillation_vibethinker-3b_fable-glint_unsloth.ipynb`, executed non-interactively with
|
|
papermill on a SLURM H100 partition (Unsloth + TRL, LoRA).
|
|
|
|
---
|
|
|
|
*Card generated from the training run's own configuration and logs by*
|
|
*`scripts/generate_hub_model_card.py`.*
|