90 lines
2.6 KiB
Markdown
90 lines
2.6 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- gpt-oss
|
|||
|
|
- mixture-of-experts
|
|||
|
|
- moe
|
|||
|
|
- reasoning
|
|||
|
|
- sft
|
|||
|
|
- science
|
|||
|
|
base_model: openai/gpt-oss
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# gpt-oss-15.5b-23E-SFT
|
|||
|
|
|
|||
|
|
A supervised fine-tuned (SFT) variant of a `gpt_oss` Mixture-of-Experts model, specialized for science-domain reasoning. The model is a pruned/customized configuration with **23 local experts** (top-4 routing) and approximately **15.5B total parameters**.
|
|||
|
|
|
|||
|
|
## Model details
|
|||
|
|
|
|||
|
|
| | |
|
|||
|
|
|---|---|
|
|||
|
|
| Architecture | `GptOssForCausalLM` (Mixture-of-Experts) |
|
|||
|
|
| Hidden size | 2880 |
|
|||
|
|
| Layers | 24 (alternating sliding / full attention) |
|
|||
|
|
| Attention heads | 64 (8 KV heads, GQA) |
|
|||
|
|
| Local experts | 23 |
|
|||
|
|
| Experts per token | 4 |
|
|||
|
|
| Vocab size | 201,088 |
|
|||
|
|
| Max position embeddings | 131,072 (YaRN, factor 32× from 4096) |
|
|||
|
|
| Sliding window | 128 |
|
|||
|
|
| dtype | bfloat16 |
|
|||
|
|
|
|||
|
|
See `config.json` for the full configuration.
|
|||
|
|
|
|||
|
|
## Training
|
|||
|
|
|
|||
|
|
This checkpoint corresponds to **step 1175** of an SFT run with the following setup:
|
|||
|
|
|
|||
|
|
- **Base**: `gpt_oss` 23-expert variant
|
|||
|
|
- **Data**: ~30k science / reasoning samples
|
|||
|
|
- **Epochs**: 5
|
|||
|
|
- **Global batch size**: 128
|
|||
|
|
- **Learning rate**: 1e-5
|
|||
|
|
- **Precision**: bfloat16
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
import torch
|
|||
|
|
|
|||
|
|
repo = "xpuenabler/gpt-oss-15.5b-23E-SFT"
|
|||
|
|
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(repo)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
repo,
|
|||
|
|
dtype=torch.bfloat16,
|
|||
|
|
device_map="auto",
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
messages = [
|
|||
|
|
{"role": "user", "content": "Explain why the sky is blue."},
|
|||
|
|
]
|
|||
|
|
inputs = tokenizer.apply_chat_template(
|
|||
|
|
messages,
|
|||
|
|
add_generation_prompt=True,
|
|||
|
|
return_tensors="pt",
|
|||
|
|
).to(model.device)
|
|||
|
|
|
|||
|
|
outputs = model.generate(inputs, max_new_tokens=512, do_sample=False)
|
|||
|
|
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
A custom `chat_template.jinja` is bundled with the tokenizer and is applied automatically by `apply_chat_template`.
|
|||
|
|
|
|||
|
|
## Files
|
|||
|
|
|
|||
|
|
- `model.safetensors` — model weights (bfloat16, ~31 GB)
|
|||
|
|
- `config.json`, `generation_config.json` — model / generation configuration
|
|||
|
|
- `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, `chat_template.jinja` — tokenizer assets
|
|||
|
|
|
|||
|
|
## Intended use & limitations
|
|||
|
|
|
|||
|
|
This is a research checkpoint intended for science-oriented reasoning experiments. It has not been aligned for safety beyond what was inherited from the base model and the SFT data, and may produce inaccurate or unsafe outputs. Evaluate carefully before any downstream use.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Released under the Apache 2.0 license, consistent with the upstream `gpt-oss` family.
|