329 lines
10 KiB
Markdown
329 lines
10 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
tags:
|
||
- text-generation
|
||
- gpt2
|
||
- causal-lm
|
||
- fine-tuned
|
||
- knowledge-repair
|
||
pipeline_tag: text-generation
|
||
model_type: gpt2
|
||
---
|
||
|
||
# FF_3.13
|
||
|
||
> **Champion model** of the FF-LLM line. A 2.02B GPT-2 architecture model fine-tuned through a multi-stage pipeline (pretraining → SFT → distillation → surgical fine-tuning → knowledge repair) for general-purpose factual question answering.
|
||
|
||
---
|
||
|
||
## Model overview
|
||
|
||
| | |
|
||
|---|---|
|
||
| **Architecture** | GPT-2 (causal LM) |
|
||
| **Parameters** | 2.02B |
|
||
| **Hidden size** | 2048 |
|
||
| **Layers** | 38 |
|
||
| **Attention heads** | 16 |
|
||
| **Vocab size** | 50,257 |
|
||
| **Tokenizer** | GPT-2 BPE |
|
||
| **Context length** | 1024 tokens |
|
||
| **Precision** | bfloat16 (also fp16/fp32 compatible) |
|
||
| **License** | Apache 2.0 |
|
||
| **Author** | francescofiamingo1 |
|
||
|
||
---
|
||
|
||
## Benchmark performance
|
||
|
||
### MMLU (Massive Multitask Language Understanding)
|
||
|
||
Evaluated with `lm-eval-harness v0.4.11`, greedy decoding.
|
||
|
||
| Split | Score |
|
||
|---|---|
|
||
| **MMLU full (14,042 items)** | **28.05%** |
|
||
| MMLU dev (285 items) | 25.61% |
|
||
|
||
### Macro-domain breakdown (MMLU full)
|
||
|
||
| Macro | Subjects | Accuracy |
|
||
|---|---|---|
|
||
| STEM | 19 subjects | 30.70% |
|
||
| Humanities | 13 subjects | 26.06% |
|
||
| Social Sciences | 12 subjects | 30.03% |
|
||
| Other (medicine, law, professional) | 13 subjects | 29.32% |
|
||
|
||
### 106-bench (custom factual benchmark)
|
||
|
||
Custom 106-prompt benchmark with strict TRUTH-list scoring:
|
||
|
||
| Category | N | Score |
|
||
|---|---|---|
|
||
| arithmetic | 5 | 5/5 (100.0%) |
|
||
| open-ended | 1 | 1/1 (100.0%) |
|
||
| person | 25 | 22/25 (88.0%) |
|
||
| science | 25 | 21/25 (84.0%) |
|
||
| geography | 25 | 15/25 (60.0%) |
|
||
| format compliance | 25 | 15/25 (60.0%) |
|
||
| **TOTAL** | **106** | **79/106 (74.5%)** |
|
||
|
||
### Improvement vs precursors
|
||
|
||
| Model | MMLU full | Δ vs FF_3.13 |
|
||
|---|---|---|
|
||
| FF_3 (base, original release) | — | — |
|
||
| FF_3.1 (post-SFT) | 26.72% | -1.33pp |
|
||
| FF_3.11 (specialized variant) | 25.20% | -2.85pp |
|
||
| **FF_3.13 (this model)** | **28.05%** | **— champion** |
|
||
|
||
---
|
||
|
||
## Training pipeline — chronological view
|
||
|
||
The model went through **7 distinct stages** of training. Below is the complete history.
|
||
|
||
### Stage 1 — Pretraining
|
||
|
||
**Architecture chosen:** GPT-2 (2.02B parameters), trained from scratch on a curated multi-source web + encyclopedic + educational corpus.
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Hardware | 8× NVIDIA RTX 5090 (24 GB each = 192 GB VRAM total) |
|
||
| Throughput | **~220,000 tokens/sec sustained** (100% GPU utilization, all 8 GPUs in parallel) |
|
||
| Framework | PyTorch + DeepSpeed ZeRO-2 |
|
||
| Precision | bfloat16 |
|
||
| **Total pretraining tokens** | **~90 billion tokens** |
|
||
| Wall-clock pretraining time | ~5 days continuous (90B / 220K tok/s ≈ 4.7 days) |
|
||
|
||
#### Pretraining data composition
|
||
|
||
The pretraining corpus was assembled from **8 distinct sources**, organized in two training modules (M1 BASE / Extra and M2 BASE / Extra) with quality-tiered weighting.
|
||
|
||
| Dataset | Module | Type | Quality weight |
|
||
|---|---|---|---|
|
||
| **FineWeb general** | M1 BASE | Web | medium |
|
||
| **FineWeb 10BT** | M1 Extra25 | Web high quality | medium |
|
||
| **FineWeb EDU** | M2 BASE | Educational | **high** |
|
||
| **FineWeb EDU extended** | M2 Extra | Educational reasoning | medium |
|
||
| **C4 EN** | M1 C4 | Web filtered | medium |
|
||
| **Wikipedia EN** | M1 BASE | Encyclopedic | low |
|
||
| **Web Clean custom** | M1 BASE / Extra | Web filtered | low |
|
||
| **News crawl** | M1 BASE | Journalistic | low |
|
||
|
||
#### Mix proportions (approximate)
|
||
|
||
- **60–65% FineWeb** (various slices: general, 10BT, EDU, EDU extended)
|
||
- **15–20% C4 EN**
|
||
- **5–10% Wikipedia EN**
|
||
- **5–10% Web Clean custom**
|
||
- **~5% News crawl**
|
||
|
||
This mix prioritizes educational content (FineWeb EDU = high weight) and high-quality web text, with encyclopedic and journalistic sources providing factual grounding.
|
||
|
||
### Stage 2 — Supervised Fine-Tuning (SFT)
|
||
|
||
**Objective:** general instruction-following + factual knowledge alignment.
|
||
|
||
**Data sources (~860K total examples):**
|
||
- **OpenHermes** (cleaned)
|
||
- **UltraChat** (cleaned)
|
||
- **WildChat** (cleaned)
|
||
- **Numina** (math reasoning)
|
||
- **OpenThoughts** (chain-of-thought)
|
||
- **Eurus** (multi-task)
|
||
|
||
Composition: ~760K core + 100K augmentation examples. Sharded under `s3://ff-llm-datasets/sft/shards_v2/`.
|
||
|
||
### Stage 3 — Direct Preference Optimization (DPO) — **REJECTED**
|
||
|
||
**Two DPO experiments were attempted and discarded:**
|
||
|
||
| DPO variant | Pairs | Result |
|
||
|---|---|---|
|
||
| v1 — WizardLM/Alpaca preferences | 38,863 | **-3pp MMLU** → rejected |
|
||
| v2 — UltraFeedback (argilla/ultrafeedback-binarized) | 60,917 | **-3pp MMLU** → rejected |
|
||
|
||
**Lesson:** DPO consistently caused MMLU regression (~-3pp) regardless of hyperparameters. **Not used in the final model.**
|
||
|
||
### Stage 4 — Distillation v3
|
||
|
||
Knowledge distillation from larger teacher models on a curated question set.
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Total questions | 108,779 |
|
||
| Source mix | hellaswag (37%), openhermes (28%), mmlu (14%), math (12%), gsm8k (7%), arc (2%), truthfulqa (1%) |
|
||
| S3 path | `s3://ff-llm-datasets/distill_v3/` |
|
||
|
||
### Stage 5 — LoRA experiments — **REJECTED**
|
||
|
||
Multiple LoRA fine-tuning attempts were tried for surgical improvements:
|
||
|
||
| LoRA experiment | Examples | Result |
|
||
|---|---|---|
|
||
| LoRA v4b (synthetic instruction) | 6,000 | marginal, not promoted |
|
||
| LoRA format-only v1/v3 | 1,779–2,092 | **catastrophic forgetting** (-3 to -4pp MMLU full) |
|
||
|
||
**Lesson:** LoRA at LR ≥ 5e-4 with template-structured data caused the model to overfit to template patterns rather than learn generalizable behavior. **Not used in the final model.**
|
||
|
||
### Stage 6 — Surgical Fine-Tuning
|
||
|
||
Targeted fine-tuning on a small curated set focused on output discipline (yes/no answers, single-letter MCQ, exact-N lists, numeric-only).
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Examples | 3,000 |
|
||
| Path | `D:\ff_llm\ff31_surgical.jsonl` |
|
||
|
||
### Stage 7 — Knowledge Repair Training (produces FF_3.13)
|
||
|
||
The decisive stage that turned FF_3.11 into FF_3.13.
|
||
|
||
**Dataset composition (16,006 total):**
|
||
|
||
| Block | Description | Examples |
|
||
|---|---|---|
|
||
| Block A | MMLU-style MCQ (multiple choice questions across diverse subjects) | 10,714 |
|
||
| Block B | Factual concise (TruthfulQA-like, <100 char answers) | 929 |
|
||
| Block D | Numeric microreasoning (arithmetic word problems with step solutions) | 3,562 |
|
||
| Validation set | held-out for monitoring | 801 |
|
||
|
||
**Training configuration:**
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Hardware | 8× NVIDIA RTX 5090 |
|
||
| Framework | DeepSpeed ZeRO-2 |
|
||
| Precision | bfloat16 |
|
||
| Optimizer | AdamW |
|
||
| Learning rate | 2.5e-6 (cosine schedule) |
|
||
| Epochs | 3 (early-stopped at step 200/357) |
|
||
| Effective batch size | (configured for 8-GPU DDP) |
|
||
| Wall-clock | ~30 min total |
|
||
|
||
**Checkpoint sweep & selection:**
|
||
|
||
| Checkpoint | MMLU full | Status |
|
||
|---|---|---|
|
||
| 1-epoch ckpt-100 | 27.47% | not selected |
|
||
| 3-epoch ckpt-50 | 27.21% | not selected |
|
||
| 3-epoch ckpt-100 | 27.86% | not selected |
|
||
| **3-epoch ckpt-150** | **28.05%** | **CHAMPION → FF_3.13** ✅ |
|
||
| 3-epoch ckpt-200 | 28.17% | rejected (+0.12pp marginal, regressed 5/6 weak domains) |
|
||
|
||
---
|
||
|
||
## Compute infrastructure summary
|
||
|
||
| Resource | Specification |
|
||
|---|---|
|
||
| GPU | 8× NVIDIA RTX 5090 (24 GB VRAM each, 192 GB total) |
|
||
| GPU utilization | ~100% sustained during training |
|
||
| Throughput (pretraining) | ~220,000 tokens/sec |
|
||
| Distributed training | DeepSpeed ZeRO-2 |
|
||
| Numerical precision | bfloat16 (training and inference) |
|
||
| Cloud provider | Vast.ai |
|
||
|
||
---
|
||
|
||
## Recommended usage
|
||
|
||
### Prompt template (Alpaca-style)
|
||
|
||
```
|
||
### System:
|
||
You are FF-LLM, a helpful assistant.
|
||
|
||
### Instruction:
|
||
<your question>
|
||
|
||
### Response:
|
||
```
|
||
|
||
### Decoding settings
|
||
|
||
- **Always use greedy decoding** (`do_sample=False`).
|
||
- Sampling has been shown to degrade factual accuracy by ~5pp on this model family.
|
||
|
||
### Quick start (transformers)
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
model_id = "francescofiamingo1/FF_3.13"
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")
|
||
|
||
prompt = """### System:
|
||
You are FF-LLM, a helpful assistant.
|
||
|
||
### Instruction:
|
||
What is the capital of France?
|
||
|
||
### Response:
|
||
"""
|
||
|
||
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
|
||
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=False)
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
```
|
||
|
||
---
|
||
|
||
## Limitations and known weaknesses
|
||
|
||
- **2B parameters** — knowledge ceiling lower than 7B+ models
|
||
- **Format compliance moderate**: 60% on strict format-discipline bench (yes/no, exact-N, single-letter)
|
||
- **Entity disambiguation weakness**: occasional "anchor entity" over-attribution (e.g., default to Edison for inventor questions)
|
||
- **Weak domains** (per qualitative analysis): mathematics, literature, music, art
|
||
- **Strong domains**: biology, geography, basic science, factual short-form QA
|
||
|
||
---
|
||
|
||
## Variant lineage
|
||
|
||
| Variant | Status | Notes |
|
||
|---|---|---|
|
||
| FF_3 | base | initial release |
|
||
| FF_3.1 | published | post-SFT, MMLU 26.72% |
|
||
| FF_3.2 | **discontinued** | early experiment, not maintained |
|
||
| FF_3.11 | published | specialized variant, MMLU 25.20%, 106-bench 71% |
|
||
| **FF_3.13** | **current champion** | knowledge repair on FF_3.11 base, MMLU 28.05% |
|
||
| FF_3.14 | **rejected** | full SFT with humanities focus, MMLU flat (no improvement) |
|
||
| SLERP t=0.10 (FF_3.13 + FF_3.11) | candidate backup | MMLU 29.10% (+0.41pp), 106-bench tie |
|
||
|
||
---
|
||
|
||
## Reproducibility
|
||
|
||
All training data shards, scripts, and intermediate checkpoints are tracked in cloud storage:
|
||
|
||
- Datasets: `s3://ff-llm-datasets/`
|
||
- Champion model: `s3://ff-llm-datasets/champions/latest/`
|
||
- Build scripts: `s3://ff-llm-datasets/ff314/build/` (includes Block E/F builders, anti-anchoring tables, philosophy seeds)
|
||
|
||
For reproduction support, contact the author.
|
||
|
||
---
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{ff_3_13_2026,
|
||
author = {francescofiamingo1},
|
||
title = {FF_3.13: a 2B GPT-2 model with knowledge repair fine-tuning},
|
||
year = {2026},
|
||
publisher = {Hugging Face},
|
||
howpublished = {\url{https://huggingface.co/francescofiamingo1/FF_3.13}}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
*Last updated: 2026-04-18*
|