初始化项目，由ModelHub XC社区提供模型

Model: Nwna/olmo3-190m-zh-v2-base Source: Original Platform
2026-07-03 20:07:33 +08:00
commit 10accdcff7
9 changed files with 239462 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,185 @@
+---
+license: apache-2.0
+language:
+- zh
+tags:
+- pretrained
+- olmo3
+- chinese
+- base-model
+- llm001
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+- Nwna/olmo3-190m-zh-v2-base-data
+---
+
+# OLMo3-190M-zh-v2-base
+
+**中文 from-scratch pretrain 教学模型**。本模型使用自训 48k 中文 BPE tokenizer 和约 **3.3907B tokens** 中文语料，从零训练 OLMo3-190M 架构 1 epoch。
+
+> 这是《零基础 AI 大模型研发训练营》第 04 讲作业模型。它是 base model，不是 instruction/chat model；适合用于中文续写、继续预训练和后续 SFT。
+
+## 来源与动机
+
+本模型的目标是完整走通小型中文 LLM 的基础预训练链路：
+
+```text
+原始中文语料
+  -> 自训 48k 中文 BPE tokenizer
+  -> 构建 3.3907B tokenized base 数据
+  -> 从零训练 OLMo3-190M base model 1 epoch
+  -> 生成评估
+  -> 推送 HuggingFace
+```
+
+这条 v2 链路没有直接复用老师旧版 tokenized 数据，而是重新训练 tokenizer、重新构建 tokenized 数据，并同步模型 config 的 `vocab_size` 与特殊 token id。
+
+## 架构
+
+基于 OLMo3-190M causal LM 架构：
+
+| 字段 | 值 |
+|---|---:|
+| `architectures` | `["Olmo3ForCausalLM"]` |
+| `hidden_size` | 768 |
+| `num_hidden_layers` | 12 |
+| `num_attention_heads` | 12 |
+| `intermediate_size` | 3072 |
+| `vocab_size` | 48,000 |
+| `max_position_embeddings` | 4096 |
+| training seq_len | 2048 |
+| total params | ~187M |
+
+## 使用
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("Nwna/olmo3-190m-zh-v2-base")
+model = AutoModelForCausalLM.from_pretrained("Nwna/olmo3-190m-zh-v2-base")
+
+inputs = tok("人工智能是", return_tensors="pt")
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=120,
+    do_sample=True,
+    temperature=0.8,
+    top_p=0.9,
+)
+print(tok.decode(outputs[0], skip_special_tokens=True))
+```
+
+### 用于继续预训练 / SFT
+
+这是 base model，没有对话能力。推荐用途是：
+
+```python
+from transformers import AutoModelForCausalLM
+
+model = AutoModelForCausalLM.from_pretrained("Nwna/olmo3-190m-zh-v2-base")
+# 继续做 continue pretrain 或作为 SFT 底座
+```
+
+## 训练配方
+
+### Tokenizer 与数据
+
+| 项目 | 值 |
+|---|---:|
+| tokenizer | `Nwna/olmo3-190m-zh-v2-tokenizer` |
+| tokenizer 类型 | 中文 Byte-level BPE |
+| vocab size | 48,000 |
+| tokenized data | `Nwna/olmo3-190m-zh-v2-base-data` |
+| actual tokens | 3,390,687,394 |
+| dtype | uint16 |
+
+数据主要来自：
+
+| 数据源 | 作用 |
+|---|---|
+| Ultra-FineWeb-zh | 提供通用中文网页文本，覆盖面广 |
+| Fineweb-Edu-Chinese-V2.2 | 提供教育类文本，提高知识密度 |
+
+### 超参
+
+| 项目 | 值 |
+|---|---:|
+| GPU | H100 |
+| precision | bf16 |
+| attention | SDPA |
+| epochs | 1.0 |
+| global steps | 12,288 |
+| per-device batch | 16 |
+| grad accumulation | 8 |
+| effective batch | 128 sequences |
+| tokens / step | 262,144 |
+| peak lr | 5e-4 |
+| min lr | 5e-5 |
+| scheduler | warmup + cosine |
+| optimizer | AdamW |
+| weight decay | 0.1，embedding / norm / bias excluded |
+
+## Loss 轨迹
+
+| step | eval_loss | eval_ppl |
+|---:|---:|---:|
+| 4,000 | 4.1429 | 63.0 |
+| 8,000 | 3.9029 | 49.5 |
+| 12,000 | 3.8042 | 44.9 |
+
+最后 100 条训练日志的 train loss 平均约 **3.797**。训练结束在 step 12,288，最后一次 eval 发生在 step 12,000。
+
+
+## 7-prompt 抽测表现
+
+使用 `src/eval.py` 的采样生成设置：
+
+```text
+max_new_tokens=120
+do_sample=True
+temperature=0.8
+top_p=0.9
+repetition_penalty=1.1
+```
+
+| Prompt | 表现 | 观察 |
+|---|---|---|
+| 人工智能是 | 🟡 | 能生成中文科普段落，但重复与套话明显 |
+| 山里有座庙 | 🟡 | 可续写，容易转成虚构的文物/寺庙说明 |
+| 今天天气不错，我准备 | 🟡 | 能续写日常文本，但上下文关系混乱 |
+| 北京大学位于 | 🔴 | 地理事实不稳定，可能答到错误地点 |
+| 四大发明是 | 🔴 | 不能稳定列出正确四大发明 |
+| 《红楼梦》人物 | 🔴 | 容易生成标题式重复或人物混杂 |
+| Python 是一种 | 🟡 | 知道与编程相关，但定义不稳定 |
+
+## 限制
+
+1. **不是 chat / instruction model**：没有经过 SFT 或 RLHF，不具备稳定对话能力。
+2. **事实可靠性有限**：190M 小模型在单轮 1 epoch 训练后仍会产生事实混淆。
+3. **采样输出会波动**：`do_sample=True` 时，同一 prompt 多次生成可能不同。
+4. **上下文控制较弱**：开放式 prompt 可能漂移到网页、列表、故事或百科腔。
+
+## License
+
+- **模型权重**：Apache-2.0
+- **训练数据**：混合来源。请根据各原始数据源许可确认下游使用边界。
+
+## 配套资源
+
+- Tokenizer: <https://huggingface.co/Nwna/olmo3-190m-zh-v2-tokenizer>
+- Tokenized data: <https://huggingface.co/datasets/Nwna/olmo3-190m-zh-v2-base-data>
+- Space demo: <https://huggingface.co/spaces/Nwna/olmo3-190m-zh-v2-base>
+- Continue model: <https://huggingface.co/Nwna/olmo3-190m-zh-v2-continue>
+
+## Citation
+
+```bibtex
+@misc{nwna-olmo3-190m-zh-v2-base,
+  title={OLMo3-190M-zh-v2-base: Chinese From-Scratch Pretrained Teaching Model},
+  author={Nwna},
+  year={2026},
+  howpublished={\url{https://huggingface.co/Nwna/olmo3-190m-zh-v2-base}},
+  note={LLM001 Course, Lecture 04 homework}
+}
+```
--- a/config.json
+++ b/config.json
@@ -0,0 +1,43 @@
+{
+  "architectures": [
+    "Olmo3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_implementation": "sdpa",
+  "bos_token_id": 2,
+  "dtype": "float32",
+  "eos_token_id": 0,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 4096,
+  "model_type": "olmo3",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 500000,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.6",
+  "use_cache": true,
+  "vocab_size": 48000
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": [
+    0
+  ],
+  "pad_token_id": 1,
+  "transformers_version": "4.57.6"
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:911b8ebddd2d7b79a8b2b3d087bc40e7dfa44c88e2a3e8c502299df9a6afe183
+size 748062408
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,36 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<think>",
+    "</think>"
+  ],
+  "bos_token": {
+    "content": "<|bos|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|unk|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,82 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|bos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|unk|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<think>",
+    "</think>"
+  ],
+  "bos_token": "<|bos|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<|unk|>"
+}
--- a/training_args.bin
+++ b/training_args.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a778cd7c168bcab0405c2127e84a652ffd118a64aabbdd8d55b0f2d66c723111
+size 5432