---
base_model: EleutherAI/pythia-410m
base_model_relation: finetune
datasets:
- yahma/alpaca-cleaned
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- pythia
- alpaca
- instruction-tuning
- full-finetune
---
# Pythia-410M step5000 Alpaca full fine-tune
> **An intentionally small, inspectable before/after experiment:** can one epoch of Alpaca SFT turn a very early Pythia checkpoint from prompt-template repetition into a recognizable instruction follower?
This is a full-parameter supervised fine-tune of the early `step5000` revision of [`EleutherAI/pythia-410m`](https://huggingface.co/EleutherAI/pythia-410m). It was trained for one epoch on 10,000 examples from [`yahma/alpaca-cleaned`](https://huggingface.co/datasets/yahma/alpaca-cleaned), using a separate fixed set of 256 examples for validation.
This repository contains a complete model checkpoint. It is not a LoRA or PEFT adapter and does not require the original checkpoint to be loaded separately.
**[Download the complete checkpoint](https://huggingface.co/shehryars715/pythia-410m-step5000-alpaca/tree/main)** · **[See all raw before/after generations](https://huggingface.co/shehryars715/pythia-410m-step5000-alpaca/blob/main/comparison_results.csv)**
## Why this checkpoint is interesting
- **A genuine early-checkpoint transformation:** the base `step5000` model mostly loops over prompt markers; after SFT it emits answer-shaped responses.
- **A full model, not an adapter:** useful for direct loading and for studying how every parameter changes during instruction tuning.
- **Small enough to explore:** 410M parameters is approachable for educational inference and analysis.
- **Reproducible:** the training split, validation split, prompt format, hyperparameters, deterministic test prompts, and decoding settings are documented.
- **Transparent about failure:** the included comparisons show improved instruction behavior alongside major factual weaknesses.
## At a glance
| Property | Value |
|---|---|
| Starting point | Pythia-410M at pretraining revision `step5000` |
| Fine-tuning data | 10,000 cleaned Alpaca examples |
| Validation data | 256 fixed held-out examples |
| Update method | Full-parameter SFT, completion-only loss |
| Training duration | 1 epoch |
| Context length | 384 tokens |
| Best use | Education, checkpoint analysis, SFT demonstrations |
| Production ready | **No** |
## Model details
- **Architecture:** GPT-NeoX causal language model
- **Base model:** `EleutherAI/pythia-410m`
- **Base revision:** `step5000`
- **Language:** primarily English
- **Task:** instruction-conditioned text generation
- **Fine-tuning method:** full-parameter supervised fine-tuning
- **Quantization:** none
- **Adapters:** none
The starting revision is only partially pretrained. Alpaca fine-tuning teaches instruction/response structure but does not replace the language pretraining missing from this early checkpoint.
## Training data
The source dataset was filtered to remove examples with empty outputs, shuffled with seed 42, and split as follows:
- First 256 shuffled examples: held-out validation set
- Next 10,000 examples: training set
Examples were converted to the original Alpaca prompt format. The target response was kept as a separate completion, with EOS appended. Loss was calculated only on completion and EOS tokens; prompt tokens supplied context but were masked from the loss.
## Training procedure
| Setting | Value |
|---|---:|
| Epochs | 1 |
| Training examples | 10,000 |
| Validation examples | 256 |
| Maximum sequence length | 384 |
| Per-device batch size | 2 |
| Gradient accumulation | 8 |
| Learning rate | 5e-5 |
| Scheduler | Cosine |
| Warmup ratio | 0.03 |
| Weight decay | 0.01 |
| Optimizer | AdamW |
| Gradient clipping | 1.0 |
| Packing | Disabled |
| Gradient checkpointing | Enabled during training |
| Seed | 42 |
BF16 was selected only on GPUs with compute capability 8.0 or newer; otherwise FP16 mixed precision was used. All model parameters were enabled for training.
## Prompt format
Without an additional input:
```text
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
```
With an additional input:
```text
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Input:
{input}
### Response:
```
## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_ID = "shehryars715/pythia-410m-step5000-alpaca"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
dtype=dtype,
).to(device)
model.eval()
instruction = "Explain photosynthesis in exactly two sentences."
prompt = (
"Below is an instruction that describes a task. Write a response that "
"appropriately completes the request.\n\n"
f"### Instruction:\n{instruction}\n\n"
"### Response:\n"
)
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.inference_mode():
output = model.generate(
**inputs,
do_sample=False,
max_new_tokens=128,
repetition_penalty=1.05,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
)
new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())
```
## Evaluation
The training notebook evaluates the untouched checkpoint and the fine-tuned model on the same 256 held-out examples. It also compares deterministic generations on six fixed prompts and includes limited rule-based checks plus a blank human-review table.
### What changed in the six-prompt comparison
| Observation | Before SFT | After SFT |
|---|---:|---:|
| Avoided the prompt-template repetition seen in the base response | 0/6 | 6/6 |
| Passed the notebook's four transparent surface-rule checks | 0/4 | 1/4 |
The useful signal is **behavioral, not capability-level**: the model learned to produce response-shaped text, but most tested answers remain incorrect. The rule checks cover only translation keywords, speed, prime-number logic, and three-item formatting.
### Side-by-side results
| Prompt | `step5000` before SFT | After full fine-tuning |
|---|---|---|
| Photosynthesis in exactly two sentences | Repeats “Write a response…” and `### Response:` | Produces two complete sentences about photosynthesis |
| Translate weather sentence into French | Repeats instruction/response markers | Produces a direct sentence response |
| Average speed: 180 km in 3 hours | Repeats the prompt and markers | Produces a concise numerical response |
| Python prime-checking function | Repeats the prompt and markers | Produces a Python function and explanation |
| Exactly three interview tips | Repeats instruction/response markers | Produces three numbered tips |
| Why leaves appear green | Repeats instruction/response markers | Produces answer-like paragraphs |
Example 1: visible instruction-following improvement
**Prompt**
> Explain photosynthesis in exactly two sentences.
**Before SFT**
> Write a response that accurately completes the request.
> `### Response:`
> *(the same template continues repeatedly)*
**After SFT**
> Photosynthesis is a process by which plants and animals convert sunlight, carbon dioxide, and other organic compounds into glucose, oxygen, and other chemical compounds. It is a complex process that involves the conversion of carbon dioxide into glucose, oxygen, and other chemical compounds.
The response now follows the requested two-sentence shape, although its claim about animals is inaccurate.
Example 2: fluent form does not guarantee correctness
**Prompt**
> Calculate the average speed for 180 km in 3 hours.
**Before SFT**
> Calculate the average speed for 180 km in 3 hours.
> `### Instruction:`
> *(the prompt and markers continue repeatedly)*
**After SFT**
> The average speed for 180 km in 3 hours is approximately 0.8 km/h.
The expected answer is 60 km/h. This is a compact example of why the checkpoint should be studied, not trusted.
Example 3: formatting learned better than content
**Prompt**
> Give exactly three concise job interview tips.
**After SFT**
> 1. **Know Your Job:** A good job interview is a great way to learn about your skills, interests, and abilities.
> 2. **Be Prepared:** Prepare your resume and cover letter to ensure that you are prepared for the interview.
> 3. **Be Prepared for the Interview:** Prepare your resume and cover letter to ensure that you are prepared for the interview.
The response has three distinguishable items, but it is redundant and not especially concise.
The complete, unedited generations are provided in [`comparison_results.csv`](https://huggingface.co/shehryars715/pythia-410m-step5000-alpaca/blob/main/comparison_results.csv). Measured baseline and final validation losses were not present in that CSV, so no loss values are invented here.
## Intended uses
- Studying the effect of instruction tuning on an early language-model checkpoint
- Demonstrating full-parameter SFT and completion-only loss
- Small-scale educational experiments with Pythia checkpoints
## Limitations and risks
- The `step5000` checkpoint is only partially pretrained.
- The model is expected to be weak in factual knowledge, reasoning, safety, formatting reliability, and multi-turn conversation.
- Outputs may be incorrect, biased, incoherent, repetitive, or unsafe.
- The model has not been evaluated sufficiently for consequential decisions or production deployment.
- The simple rule-based checks in the training notebook are not substitutes for broad human evaluation.
Users should validate outputs for their domain and add appropriate safeguards before any interactive use.
## License and attribution
The Pythia base model is published under Apache-2.0. At the time this card was written, the Hugging Face metadata for `yahma/alpaca-cleaned` displayed CC BY 4.0, while text inside its dataset card stated CC BY-NC 4.0. Review the current upstream model and dataset terms before choosing a license for or redistributing this fine-tuned checkpoint.
## Citation
```bibtex
@article{biderman2023pythia,
title={Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling},
author={Biderman, Stella and Schoelkopf, Hailey and Anthony, Quentin and others},
journal={Proceedings of the 40th International Conference on Machine Learning},
year={2023}
}
```
```bibtex
@misc{alpaca,
author={Taori, Rohan and Gulrajani, Ishaan and Zhang, Tianyi and others},
title={Stanford Alpaca: An Instruction-following LLaMA Model},
year={2023},
howpublished={\url{https://github.com/tatsu-lab/stanford_alpaca}}
}
```