初始化项目，由ModelHub XC社区提供模型

Model: ynanxiu/olmo3-190M-zh-full Source: Original Platform
2026-07-23 21:18:46 +08:00
commit 757fcaeab6
8 changed files with 239315 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,141 @@
+---
+language: zh
+license: apache-2.0
+library_name: transformers
+tags:
+- olmo3
+- chinese
+- pretrained
+- text-generation
+- rtx4060
+pipeline_tag: text-generation
+model-index:
+- name: OLMo3-190M-zh-full
+  results:
+  - task:
+      type: text-generation
+    dataset:
+      name: Chinese Pretraining Data
+      type: llm101-olmo3-zh-demo-data
+    metrics:
+    - type: eval_loss
+      value: 3.624
+      name: Eval Loss
+---
+
+# OLMo3-190M-zh-full
+
+**从零开始、在 RTX 4060 8GB 上预训练的 190M 中文语言模型。**
+
+## 模型概述
+
+这是一个基于 OLMo3 架构的 **190M 参数中文预训练模型**，完全从随机初始化（from scratch）开始训练，未依赖任何已有预训练权重。
+
+- 🏗️ **架构**: OLMo3（Sliding Window Attention + Full Attention 混合）
+- 📏 **参数**: 187M 总参数（113M 非嵌入参数）
+- 🌐 **语言**: 中文
+- 📚 **训练数据**: `cmz1024/llm101-olmo3-zh-demo-data`（~93 万条中文序列）
+- 🖥️ **硬件**: NVIDIA RTX 4060 8GB（本地训练）
+- ⏱️ **训练耗时**: ~47 小时
+
+## 模型架构
+
+| 参数 | 值 |
+|------|-----|
+| hidden_size | 768 |
+| num_layers | 12 |
+| num_heads | 12 |
+| intermediate_size | 3072 |
+| vocab_size | 48,000 |
+| max_position_embeddings | 2048 |
+| 注意力机制 | Sliding Window（每 4 层中 3 层）+ Full Attention（1 层） |
+
+## 训练细节
+
+| 配置 | 值 |
+|------|-----|
+| 优化器 | AdamW (β1=0.9, β2=0.95) |
+| 学习率 | 5e-4（cosine decay） |
+| Warmup | 2%（~320 steps） |
+| 序列长度 | 2048 |
+| Batch Size | 1 × 128 grad_accum（等效 128） |
+| 精度 | bf16 |
+| 总步数 | 7,675 |
+| Eval Loss | **3.624** |
+
+### Loss 曲线
+
+```
+Step   500:  eval_loss=5.307
+Step  1000:  eval_loss=4.534
+Step  1500:  eval_loss=4.243
+Step  2000:  eval_loss=4.087
+Step  2500:  eval_loss=3.986
+Step  3000:  eval_loss=3.906
+Step  3500:  eval_loss=3.841
+Step  4000:  eval_loss=3.790
+Step  4500:  eval_loss=3.747
+Step  5000:  eval_loss=3.709
+Step  5500:  eval_loss=3.678
+Step  6000:  eval_loss=3.654
+Step  6500:  eval_loss=3.637
+Step  7000:  eval_loss=3.627
+Step  7500:  eval_loss=3.624
+```
+
+初始 loss 10.92 → 最终 3.624，**下降 66.8%**。
+
+## 使用方法
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model = AutoModelForCausalLM.from_pretrained(
+    "ynanxiu/olmo3-190M-zh-full",
+    torch_dtype=torch.bfloat16,
+    attn_implementation="sdpa"
+).to("cuda")
+
+tokenizer = AutoTokenizer.from_pretrained("ynanxiu/olmo3-190M-zh-full")
+
+prompt = "从前有座山，山里有座庙，"
+inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=100,
+    do_sample=True,
+    temperature=0.8,
+    top_p=0.9,
+)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## 生成示例
+
+**Prompt**: `从前有座山，山里有座庙，`
+
+**生成**: 庙里有一座庙宇，庙里有一棵大树。有一天早晨起来，发现一棵树上有个苹果树...两个大猴子在树下各捉一个，猴子在树上捉一个...
+
+**Prompt**: `人工智能是`
+
+**生成**: 人工智能是未来的趋势，它给人类带来了新的机会。在人工智能领域，人工智能的潜力是非常大的...
+
+## 限制与用途
+
+- ✅ 适用于：中文文本生成、续写、作为下游任务的基座模型
+- ⚠️ 注意：这是基础预训练模型，未经过 SFT/RLHF，输出为续写风格而非对话风格
+- ⚠️ 模型可能产生重复或不一致的内容（190M 参数规模限制）
+- 🔧 建议：可在此基础上进行持续预训练或 SFT 以获得更好效果
+
+## 环境足迹
+
+- 硬件：NVIDIA RTX 4060 8GB（消费级 GPU）
+- 训练功耗：~115W × 47h ≈ 5.4 kWh
+- 训练方式：本地训练，无云计算碳排放
+
+## 相关模型
+
+- [OLMo3-190M-zh-continue](https://huggingface.co/ynanxiu/olmo3-190M-zh-continue)：22M Nano 版本的持续预训练模型
+- 训练代码基于 [openmind-llm01](https://github.com/your-org/openmind-llm01) 课程项目
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,6 @@
+{% for message in messages %}{% if message['role'] == 'assistant' %}{% generation %}{{ '<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+' }}{% endgeneration %}{% else %}{{ '<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,44 @@
+{
+  "architectures": [
+    "Olmo3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "dtype": "bfloat16",
+  "eos_token_id": 0,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 4096,
+  "model_type": "olmo3",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.9.0",
+  "use_cache": false,
+  "vocab_size": 48000
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": [
+    0
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 1,
+  "transformers_version": "5.9.0",
+  "use_cache": true
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be912095edd2a92ab729313fa3bec20e560baa7cd7f2a6676392a6811ddac446
+size 374038864
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,16 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|bos|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 4096,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|unk|>"
+}