初始化项目，由ModelHub XC社区提供模型

Model: KiteFishAI/Minnow-Math-1.5B Source: Original Platform
2026-07-14 21:14:52 +08:00
commit ca968470b1
8 changed files with 277386 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,132 @@
+---
+license: mit
+language:
+  - en
+tags:
+  - causal-lm
+  - scientific-language-model
+  - mathematics
+  - arxiv
+  - research
+library_name: transformers
+---
+
+# Minnow-Math-1.5B
+
+**Minnow-Math-1.5B** is a ~1.5B parameter decoder-only transformer trained from scratch on raw arXiv LaTeX sources across mathematics, computer science, and theoretical physics.
+
+📄 **Paper:** https://arxiv.org/abs/2602.17288  
+💻 **Github:** https://github.com/kitefishai/Minnow-Math-1.5B
+
+This is a **base scientific language model** (not instruction-tuned).
+
+## Overview
+
+Minnow-Math-1.5B explores what it takes to train a domain-specialized scientific language model directly from structured LaTeX archives.
+
+**Training Scale**
+- ~52B pretraining tokens  
+- ~5B additional post-training tokens  
+- ~200GB processed scientific corpus  
+- LLaMA-compatible tokenizer (~102k vocab)  
+- 2× NVIDIA A100 (80GB) GPUs  
+- 24 experimental training runs  
+
+The focus of this project is *scientific language modeling robustness*, not benchmark optimization.
+
+## Model Architecture
+
+- 24 Transformer layers  
+- Hidden size: 2048  
+- FFN size: 5504  
+- 16 attention heads  
+- Context length: 4096 (trained at 768 tokens)  
+- Dense LLaMA-style architecture  
+
+**Optimization**
+- AdamW  
+- Learning rate: 2e-4  
+- Warmup: 500 steps  
+- Weight decay: 0.1  
+- Gradient accumulation: 32  
+- bf16 mixed precision  
+- Gradient checkpointing enabled  
+
+**Validation Perplexity:** ~4.2 (held-out scientific corpus)
+
+## Intended Use
+
+Minnow-Math-1.5B is suitable for:
+
+- Scientific text modeling research  
+- Mathematical language modeling experiments  
+- Pretraining initialization for domain fine-tuning  
+- Tokenization and symbolic modeling research  
+- Studying LaTeX structure modeling  
+
+It is **not optimized for:**
+
+- Instruction following  
+- Chat-based applications  
+- General conversational AI  
+- Benchmark leaderboard performance  
+
+## Performance Notes
+
+This model was trained under moderate compute constraints and without instruction tuning or alignment stages.
+
+Observed characteristics:
+
+- Strong familiarity with scientific writing style  
+- Stable LaTeX structural modeling  
+- Reasonable symbolic fluency  
+- Limited reasoning depth  
+- Low downstream benchmark accuracy without fine-tuning  
+
+Performance improves significantly with supervised fine-tuning (SFT), LoRA adaptation, or domain-specific instruction tuning.
+
+## Limitations
+
+- Not instruction-tuned  
+- No RLHF or preference alignment  
+- Trained at 768-token sequence length  
+- Domain restricted to selected arXiv categories  
+- Not optimized for reasoning benchmarks  
+- General NLP benchmark scores may be low  
+
+This release is intended primarily for research and experimentation.
+
+## Example Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_id = "KiteFishAI/Minnow-Math-1.5B"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id)
+
+prompt = "Prove that the sum of two continuous functions is continuous."
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(**inputs, max_new_tokens=200)
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+
+```
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```
+@article{kitefish_a1_2026,
+  title={KiteFish-A1: Training a Scientific Language Model from Raw LaTeX Archives},
+  author={...},
+  year={2026},
+  eprint={2602.17288},
+  archivePrefix={arXiv}
+}
+```