--- tags: - gguf - llama.cpp - unsloth - llama - llama3.2 - distillation - reasoning - fine-tuning license: llama3.2 datasets: - angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k language: - en base_model: - unsloth/Llama-3.2-3B-Instruct-bnb-4bit pipeline_tag: text-generation library_name: transformers --- # Llama 3.2 3B — Claude Reasoning Distill This model was a second attempt at reasoning distillation, with several fixes from the 1B run — but the core approach was still wrong. **1. Same root problem: SFT copies style, not capability** - GRPO is the right approach **2. Dataset truncation caused the stopping problem** The training dataset (`angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k`) averages ~1,954 tokens per example, with p90 assistant responses alone hitting ~1,760 tokens. Trained at `seq_len=2048`, a significant portion of examples were silently truncated — cutting off the `<|eot_id|>` end-of-turn token before it could be written. The model learned from many examples that responses don't need to end. This is a dataset fit problem, not a model problem. **3. Wrong EOS token at inference** Llama 3 has two EOS-like tokens. `tokenizer.eos_token_id` returns `128001` (`<|end_of_text|>`), but the model generates `128009` (`<|eot_id|>`) to end a turn. The default `model.generate()` call never passes `128009`, so generation runs until `max_new_tokens`. This compounds the truncation issue above. **Same Fix as 1B if you're using this model:** ```python model.generate( input_ids=inputs, eos_token_id=[128001, 128009], max_new_tokens=512, repetition_penalty=1.3, no_repeat_ngram_size=6, ) ``` For Ollama, add to your Modelfile: ``` PARAMETER stop "<|eot_id|>" PARAMETER stop "<|end_of_text|>" ``` --- An updated attempt at distilling Claude Opus 4.6/4.7 reasoning traces into a small-form-factor model. The predecessor [Llama 3.2 1B Claude Opus Reasoning Distill](https://huggingface.co/codestrate/Llama3.2-1B-Claude-Opus-Reasoning-Distill) demonstrated that a 1B model could adopt `` blocks but suffered from echolalia and a GSM8K regression. This run addresses the two root causes identified from that experiment: 1. **Capacity** — 3B sits closer to the parameter floor where structured reasoning adoption is viable, as seen in models like [Gemma 4 E2B-IT](https://huggingface.co/google/gemma-4-E2B-it) and [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) (which has `` baked into pretraining) 2. **Token boundaries** — `` and `` are registered as special tokens (vocab 128256 → 128258) with trained embeddings, giving the model a hard mode boundary instead of treating them as plain text 3. **Training on Reponses Only** - Unlike 1B run, I used the `train_on_responses_only` from `unsloth` to mask out user inputs to have a accuracy increase in multi-turn conversational fine tuning. > **Benchmarks will not be available.** --- ## Model Details | Field | Value | |---|---| | **Base model** | [`unsloth/Llama-3.2-3B-Instruct-bnb-4bit`](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-bnb-4bit) | | **Model type** | Causal LM — LoRA adapter (PEFT) on Llama-3.2-3B-Instruct | | **Language** | English | | **License** | [Meta Llama 3.2 Community License](https://www.llama.com/llama3_2/license/) | | **Training framework** | Unsloth + TRL SFTTrainer | | **Hardware** | Tesla T4 (Kaggle) | | **Max sequence length** | 2048 | --- ## Intended Use Generating step-by-step reasoning traces (`` blocks) followed by final answers across a broad range of instruction-following tasks. Useful for studying how reasoning distillation scales to sub-4B models and how registered thinking tokens affect small-model behaviour. **Not intended for:** production use, mathematical proofs requiring reliability, or replacing a larger reasoning model. Benchmark regressions vs base are expected until verified otherwise. --- ## How to Get Started ### From the adapter The LoRA adapter is available separately — load it on top of the base model without downloading the full merged weights. > **Important:** load the tokenizer from the adapter directory, not the base model. The adapter tokenizer carries the correct 128258-token vocabulary with ``/`` baked in. Using the base model tokenizer (128256) will cause an embedding dimension mismatch. ```python from unsloth import FastLanguageModel from transformers import AutoTokenizer, TextStreamer from peft import PeftModel ADAPTER_PATH = "codestrate/Llama3.2-3B-Claude-Reasoning-Distill" model, _ = FastLanguageModel.from_pretrained( model_name="unsloth/Llama-3.2-3B-Instruct-bnb-4bit", load_in_4bit=True, max_seq_length=2048, ) tokenizer = AutoTokenizer.from_pretrained(ADAPTER_PATH) # vocab=128258 model.resize_token_embeddings(len(tokenizer)) model = PeftModel.from_pretrained(model, ADAPTER_PATH) FastLanguageModel.for_inference(model) SYSTEM_PROMPT = "You are a helpful assistant. Think step by step inside ... before giving your final answer." messages = [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": "Write a Python function to check if a number is prime."}, ] inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda") streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) _ = model.generate( input_ids=inputs, streamer=streamer, max_new_tokens=1024, temperature=0.7, min_p=0.1, repetition_penalty=1.3, no_repeat_ngram_size=6, use_cache=True, ) ``` ### From GGUF (Ollama / LM Studio) A Modelfile is included for Ollama. For direct use: ``` ollama run hf.co/codestrate/Llama3.2-3B-Claude-Reasoning-Distill:Q4_K_M ``` --- ## Training Details ### Dataset [`angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k`](https://huggingface.co/datasets/angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k) — `instruct_train.jsonl` split (full instruct + reasoning, ~7,700 examples). Data already in OpenAI messages format; mapped directly through `apply_chat_template` with no additional preprocessing. The previous 1B run used only the `coding` + `math` categories (~2,000 examples). This run uses the full instruct split for broader coverage. ### Hyperparameters | Parameter | Value | |---|---| | LoRA Rank / Alpha | 32 / 64 | | Target Modules | All | | Sequence Length | 2048 | | Effective Batch | 16 (2 × grad_accum 8) | | Steps | 904 (~2 epochs) | | Learning Rate | 1e-4 / cosine | | Warmup Steps | 50 | | Optimizer | adamw_8bit | | Weight Decay | 0.01 | | Precision | bfloat16 | ### Loss Curve ![Loss Curve](loss_curve_with_rolling_average.png) | Step | Loss | Step | Loss | Step | Loss | |---|---|---|---|---|---| | 50 | 2.1372 | 350 | 1.8798 | 650 | 1.7567 | | 100 | 1.9597 | 400 | 1.8512 | 700 | 1.7530 | | 150 | 1.9251 | 450 | 1.8493 | 750 | **1.7391** | | 200 | 1.8972 | 500 | 1.7670 | 800 | 1.7709 | | 250 | 1.8891 | 550 | 1.7707 | 850 | 1.7401 | | 300 | 1.8738 | 600 | 1.7668 | 900 | 1.7598 | Drop: 2.14 → 1.74 (~0.40 absolute). Visible cross-epoch improvement at step ~452 (−0.082). Plateau reached in epoch 2 from step 750 — a third epoch would not have been beneficial on this dataset. --- ## Known Limitations - **Benchmarks not yet available** — results will be added when the evaluation runs complete - **Echolalia / repetition** — reduced vs the 1B run due to special token boundaries, but not eliminated; `repetition_penalty=1.3` and `no_repeat_ngram_size=6` are recommended at inference (needs more testing) - **System prompt required** — without the `...` contract in the system prompt, the model may not cleanly transition from reasoning block to final answer - **Not a production model** — a research artefact studying reasoning distillation at sub-4B scale --- ## Available Files | File | Format | Use | |---|---|---| | `Llama-3.2-3B-Claude-Reasoning-Distill.Q4_K_M.gguf` | GGUF Q4_K_M | LM Studio / Ollama (recommended) | | `Llama-3.2-3B-Claude-Reasoning-Distill.Q8_0.gguf` | GGUF Q8 | Higher fidelity inference (near lossless; still lightweight)| | `Llama-3.2-3B-Claude-Reasoning-Distill.F16.gguf` | GGUF F16 | Full precision GGUF | | Adapter (`adapter_model.safetensors`) | LoRA adapter | PEFT inference / further fine-tuning | --- ## Framework Versions - Python 3.12.13 - Unsloth 2026.5.8 - PEFT 0.19.1 - TRL 0.24.0 - PyTorch 2.10.0+cu128 - Transformers 4.47.1 --- *Predecessor: [Llama3.2-1B-Claude-Opus-Reasoning-Distill](https://huggingface.co/codestrate/Llama3.2-1B-Claude-Opus-Reasoning-Distill)* *Trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth)* [](https://github.com/unslothai/unsloth)