133 lines
3.4 KiB
Markdown
133 lines
3.4 KiB
Markdown
---
|
||
license: mit
|
||
language:
|
||
- en
|
||
tags:
|
||
- causal-lm
|
||
- scientific-language-model
|
||
- mathematics
|
||
- arxiv
|
||
- research
|
||
library_name: transformers
|
||
---
|
||
|
||
# Minnow-Math-1.5B
|
||
|
||
**Minnow-Math-1.5B** is a ~1.5B parameter decoder-only transformer trained from scratch on raw arXiv LaTeX sources across mathematics, computer science, and theoretical physics.
|
||
|
||
📄 **Paper:** https://arxiv.org/abs/2602.17288
|
||
💻 **Github:** https://github.com/kitefishai/Minnow-Math-1.5B
|
||
|
||
This is a **base scientific language model** (not instruction-tuned).
|
||
|
||
## Overview
|
||
|
||
Minnow-Math-1.5B explores what it takes to train a domain-specialized scientific language model directly from structured LaTeX archives.
|
||
|
||
**Training Scale**
|
||
- ~52B pretraining tokens
|
||
- ~5B additional post-training tokens
|
||
- ~200GB processed scientific corpus
|
||
- LLaMA-compatible tokenizer (~102k vocab)
|
||
- 2× NVIDIA A100 (80GB) GPUs
|
||
- 24 experimental training runs
|
||
|
||
The focus of this project is *scientific language modeling robustness*, not benchmark optimization.
|
||
|
||
## Model Architecture
|
||
|
||
- 24 Transformer layers
|
||
- Hidden size: 2048
|
||
- FFN size: 5504
|
||
- 16 attention heads
|
||
- Context length: 4096 (trained at 768 tokens)
|
||
- Dense LLaMA-style architecture
|
||
|
||
**Optimization**
|
||
- AdamW
|
||
- Learning rate: 2e-4
|
||
- Warmup: 500 steps
|
||
- Weight decay: 0.1
|
||
- Gradient accumulation: 32
|
||
- bf16 mixed precision
|
||
- Gradient checkpointing enabled
|
||
|
||
**Validation Perplexity:** ~4.2 (held-out scientific corpus)
|
||
|
||
## Intended Use
|
||
|
||
Minnow-Math-1.5B is suitable for:
|
||
|
||
- Scientific text modeling research
|
||
- Mathematical language modeling experiments
|
||
- Pretraining initialization for domain fine-tuning
|
||
- Tokenization and symbolic modeling research
|
||
- Studying LaTeX structure modeling
|
||
|
||
It is **not optimized for:**
|
||
|
||
- Instruction following
|
||
- Chat-based applications
|
||
- General conversational AI
|
||
- Benchmark leaderboard performance
|
||
|
||
## Performance Notes
|
||
|
||
This model was trained under moderate compute constraints and without instruction tuning or alignment stages.
|
||
|
||
Observed characteristics:
|
||
|
||
- Strong familiarity with scientific writing style
|
||
- Stable LaTeX structural modeling
|
||
- Reasonable symbolic fluency
|
||
- Limited reasoning depth
|
||
- Low downstream benchmark accuracy without fine-tuning
|
||
|
||
Performance improves significantly with supervised fine-tuning (SFT), LoRA adaptation, or domain-specific instruction tuning.
|
||
|
||
## Limitations
|
||
|
||
- Not instruction-tuned
|
||
- No RLHF or preference alignment
|
||
- Trained at 768-token sequence length
|
||
- Domain restricted to selected arXiv categories
|
||
- Not optimized for reasoning benchmarks
|
||
- General NLP benchmark scores may be low
|
||
|
||
This release is intended primarily for research and experimentation.
|
||
|
||
## Example Usage
|
||
|
||
```python
|
||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
import torch
|
||
|
||
model_id = "KiteFishAI/Minnow-Math-1.5B"
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||
|
||
prompt = "Prove that the sum of two continuous functions is continuous."
|
||
inputs = tokenizer(prompt, return_tensors="pt")
|
||
|
||
with torch.no_grad():
|
||
outputs = model.generate(**inputs, max_new_tokens=200)
|
||
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
|
||
```
|
||
|
||
## Citation
|
||
|
||
If you use this model in your research, please cite:
|
||
|
||
```
|
||
@article{kitefish_a1_2026,
|
||
title={KiteFish-A1: Training a Scientific Language Model from Raw LaTeX Archives},
|
||
author={...},
|
||
year={2026},
|
||
eprint={2602.17288},
|
||
archivePrefix={arXiv}
|
||
}
|
||
```
|