87 lines
2.8 KiB
Markdown
87 lines
2.8 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model:
|
|
- unsloth/Qwen3-8B
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- unsloth
|
|
- lora
|
|
- trl
|
|
- sft
|
|
---
|
|
|
|
# Qwen3-8B-SFT-Fable5
|
|
|
|
A LoRA fine-tune of [`unsloth/Qwen3-8B`](https://huggingface.co/unsloth/Qwen3-8B), supervised fine-tuned on `ermiaazarkhalili/Fable-5-Complete-2M-Clean` (private).
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| **Base model** | [`unsloth/Qwen3-8B`](https://huggingface.co/unsloth/Qwen3-8B) |
|
|
| **Architecture** | `Qwen3ForCausalLM` |
|
|
| **Parameters** | 8.2B |
|
|
| **Training data** | `ermiaazarkhalili/Fable-5-Complete-2M-Clean` (private) |
|
|
| **Method** | LoRA supervised fine-tuning via [Unsloth](https://github.com/unslothai/unsloth) + [TRL](https://github.com/huggingface/trl) |
|
|
| **License** | `apache-2.0` (inherited from the base model) |
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "ermiaazarkhalili/Qwen3-8B-SFT-Fable5"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, dtype='auto', device_map='auto')
|
|
|
|
messages = [{"role": "user", "content": "Explain gradient checkpointing in two sentences."}]
|
|
inputs = tokenizer.apply_chat_template(
|
|
messages, add_generation_prompt=True, return_tensors='pt'
|
|
).to(model.device)
|
|
|
|
outputs = model.generate(inputs, max_new_tokens=256)
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
## Training configuration
|
|
|
|
| Setting | Value |
|
|
| --- | --- |
|
|
| LoRA rank (r) | 16 |
|
|
| LoRA alpha | 16 |
|
|
| Learning rate | 0.0002 |
|
|
| Epochs | 2 |
|
|
| Effective batch size | 8 (1 x 8 grad accum) |
|
|
| Max sequence length | 4096 |
|
|
| Base precision | 4-bit (QLoRA) |
|
|
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
|
|
|
|
## Observed training loss
|
|
|
|
Measured from our SLURM logs for this configuration. These are training-loss
|
|
observations only — no downstream benchmark evaluation has been run on this
|
|
model, so they should not be read as a quality claim.
|
|
|
|
| SLURM job | Steps | First loss | Final loss |
|
|
| --- | --- | --- | --- |
|
|
| `53213548` | 94,254 | 0.9521 | 0.7824 |
|
|
|
|
## Limitations
|
|
|
|
- No benchmark evaluation has been run on this checkpoint. The only reported
|
|
numbers are training-loss observations.
|
|
- Inherits the biases, knowledge cutoff and failure modes of the base model.
|
|
- Fine-tuned on a single instruction-following dataset; behaviour outside that
|
|
distribution is untested.
|
|
- LoRA adapters were merged into the base weights, so the merged model cannot
|
|
be detached from this fine-tune.
|
|
|
|
## Reproducing
|
|
|
|
Trained by `notebooks/fable_distillation_qwen3-8b_fable_unsloth.ipynb`, executed non-interactively with
|
|
papermill on a SLURM H100 partition (Unsloth + TRL, LoRA).
|
|
|
|
---
|
|
|
|
*Card generated from the training run's own configuration and logs by*
|
|
*`scripts/generate_hub_model_card.py`.*
|