b0c3fd01ef
Model: ermiaazarkhalili/FastContext-4B-SFT_base-SFT-Fable5-Glint Source: Original Platform
96 lines
3.4 KiB
Markdown
96 lines
3.4 KiB
Markdown
---
|
|
base_model:
|
|
- microsoft/FastContext-1.0-4B-SFT
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- unsloth
|
|
- lora
|
|
- trl
|
|
- sft
|
|
---
|
|
|
|
# FastContext-4B-SFT_base-SFT-Fable5-Glint
|
|
|
|
A LoRA fine-tune of `microsoft/FastContext-1.0-4B-SFT` (no longer available on the Hub), supervised fine-tuned on `ermiaazarkhalili/Fable-5-Glint-Clean` (private).
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| **Base model** | `microsoft/FastContext-1.0-4B-SFT` (no longer available on the Hub) |
|
|
| **Architecture** | `Qwen3ForCausalLM` |
|
|
| **Parameters** | 4.0B |
|
|
| **Training data** | `ermiaazarkhalili/Fable-5-Glint-Clean` (private) |
|
|
| **Method** | LoRA supervised fine-tuning via [Unsloth](https://github.com/unslothai/unsloth) + [TRL](https://github.com/huggingface/trl) |
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "ermiaazarkhalili/FastContext-4B-SFT_base-SFT-Fable5-Glint"
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, dtype='auto', device_map='auto')
|
|
|
|
messages = [{"role": "user", "content": "Explain gradient checkpointing in two sentences."}]
|
|
inputs = tokenizer.apply_chat_template(
|
|
messages, add_generation_prompt=True, return_tensors='pt'
|
|
).to(model.device)
|
|
|
|
outputs = model.generate(inputs, max_new_tokens=256)
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
```
|
|
|
|
## Training configuration
|
|
|
|
| Setting | Value |
|
|
| --- | --- |
|
|
| LoRA rank (r) | 16 |
|
|
| LoRA alpha | 16 |
|
|
| Learning rate | 0.0002 |
|
|
| Epochs | 3 |
|
|
| Effective batch size | 8 (2 x 4 grad accum) |
|
|
| Max sequence length | 4096 |
|
|
| Base precision | 4-bit (QLoRA) |
|
|
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
|
|
|
|
## Held-out evaluation
|
|
|
|
Next-token accuracy on a deterministic held-out split of `ermiaazarkhalili/Fable-5-Glint-Clean` (n = 199 samples), scored against the base model [`microsoft/FastContext-1.0-4B-SFT`](https://huggingface.co/microsoft/FastContext-1.0-4B-SFT). Assistant tokens only; both models are scored identically.
|
|
|
|
| Metric | Base | This model | Δ |
|
|
| --- | ---: | ---: | ---: |
|
|
| Top-1 accuracy | 0.5704 | 0.6966 | +0.1261 |
|
|
| Top-5 accuracy | 0.8203 | 0.9164 | +0.0961 |
|
|
|
|
A delta measures how far fine-tuning moved this model from its own starting point; it is not a ranking against other models, which start from different baselines.
|
|
|
|
## Observed training loss
|
|
|
|
Measured from our SLURM logs for this configuration. These are training-loss
|
|
observations only — see the held-out evaluation above for measured accuracy.
|
|
|
|
| SLURM job | Steps | First loss | Final loss |
|
|
| --- | --- | --- | --- |
|
|
| `45987981` | 1,554 | 1.3622 | 0.8805 |
|
|
| `46020980` | 1,554 | 1.3631 | 0.8779 |
|
|
|
|
## Limitations
|
|
|
|
- No benchmark evaluation has been run on this checkpoint. The only reported
|
|
numbers are training-loss observations.
|
|
- Inherits the biases, knowledge cutoff and failure modes of the base model.
|
|
- Fine-tuned on a single instruction-following dataset; behaviour outside that
|
|
distribution is untested.
|
|
- LoRA adapters were merged into the base weights, so the merged model cannot
|
|
be detached from this fine-tune.
|
|
|
|
## Reproducing
|
|
|
|
Trained by `notebooks/fable_distillation_fastcontext-4b-sft_fable-glint_unsloth.ipynb`, executed non-interactively with
|
|
papermill on a SLURM H100 partition (Unsloth + TRL, LoRA).
|
|
|
|
---
|
|
|
|
*Card generated from the training run's own configuration and logs by*
|
|
*`scripts/generate_hub_model_card.py`.*
|