初始化项目，由ModelHub XC社区提供模型

Model: KiteFishAI/Minnow-Math-1.5B
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,7 @@
+global_step1000
+latest
+rng_state_0.pth
+rng_state_1.pth
+trainer_state.json
+training_args.bin
+zero_to_fp32.py

README.md

@@ -0,0 +1,132 @@
+---
+license: mit
+language:
+  - en
+tags:
+  - causal-lm
+  - scientific-language-model
+  - mathematics
+  - arxiv
+  - research
+library_name: transformers
+---
+
+# Minnow-Math-1.5B
+
+**Minnow-Math-1.5B** is a ~1.5B parameter decoder-only transformer trained from scratch on raw arXiv LaTeX sources across mathematics, computer science, and theoretical physics.
+
+📄 **Paper:** https://arxiv.org/abs/2602.17288  
+💻 **Github:** https://github.com/kitefishai/Minnow-Math-1.5B
+
+This is a **base scientific language model** (not instruction-tuned).
+
+## Overview
+
+Minnow-Math-1.5B explores what it takes to train a domain-specialized scientific language model directly from structured LaTeX archives.
+
+**Training Scale**
+- ~52B pretraining tokens  
+- ~5B additional post-training tokens  
+- ~200GB processed scientific corpus  
+- LLaMA-compatible tokenizer (~102k vocab)  
+- 2× NVIDIA A100 (80GB) GPUs  
+- 24 experimental training runs  
+
+The focus of this project is *scientific language modeling robustness*, not benchmark optimization.
+
+## Model Architecture
+
+- 24 Transformer layers  
+- Hidden size: 2048  
+- FFN size: 5504  
+- 16 attention heads  
+- Context length: 4096 (trained at 768 tokens)  
+- Dense LLaMA-style architecture  
+
+**Optimization**
+- AdamW  
+- Learning rate: 2e-4  
+- Warmup: 500 steps  
+- Weight decay: 0.1  
+- Gradient accumulation: 32  
+- bf16 mixed precision  
+- Gradient checkpointing enabled  
+
+**Validation Perplexity:** ~4.2 (held-out scientific corpus)
+
+## Intended Use
+
+Minnow-Math-1.5B is suitable for:
+
+- Scientific text modeling research  
+- Mathematical language modeling experiments  
+- Pretraining initialization for domain fine-tuning  
+- Tokenization and symbolic modeling research  
+- Studying LaTeX structure modeling  
+
+It is **not optimized for:**
+
+- Instruction following  
+- Chat-based applications  
+- General conversational AI  
+- Benchmark leaderboard performance  
+
+## Performance Notes
+
+This model was trained under moderate compute constraints and without instruction tuning or alignment stages.
+
+Observed characteristics:
+
+- Strong familiarity with scientific writing style  
+- Stable LaTeX structural modeling  
+- Reasonable symbolic fluency  
+- Limited reasoning depth  
+- Low downstream benchmark accuracy without fine-tuning  
+
+Performance improves significantly with supervised fine-tuning (SFT), LoRA adaptation, or domain-specific instruction tuning.
+
+## Limitations
+
+- Not instruction-tuned  
+- No RLHF or preference alignment  
+- Trained at 768-token sequence length  
+- Domain restricted to selected arXiv categories  
+- Not optimized for reasoning benchmarks  
+- General NLP benchmark scores may be low  
+
+This release is intended primarily for research and experimentation.
+
+## Example Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_id = "KiteFishAI/Minnow-Math-1.5B"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id)
+
+prompt = "Prove that the sum of two continuous functions is continuous."
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(**inputs, max_new_tokens=200)
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+
+```
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```
+@article{kitefish_a1_2026,
+  title={KiteFish-A1: Training a Scientific Language Model from Raw LaTeX Archives},
+  author={...},
+  year={2026},
+  eprint={2602.17288},
+  archivePrefix={arXiv}
+}
+```

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100000,
+  "dtype": "bfloat16",
+  "eos_token_id": 100001,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 5504,
+  "max_position_embeddings": 4096,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 16,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.1.0",
+  "use_cache": false,
+  "vocab_size": 102400
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 100000,
+  "eos_token_id": 100001,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.1.0",
+  "use_cache": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ff0b84be573afae45893dcfab3082b07ebafc3d9faa790230c49b506714fb8e
+size 3267588880

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "is_local": false,
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "TokenizersBackend",
+  "truncation_side": "right",
+  "unk_token": "<unk>"
+}