初始化项目，由ModelHub XC社区提供模型

Model: ynanxiu/olmo3-190M-zh-sft Source: Original Platform
2026-10-31 19:12:05 +08:00
commit ed71c7552e
8 changed files with 239308 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,136 @@
+---
+language:
+  - zh
+tags:
+  - olmo3
+  - chinese
+  - sft
+  - chat
+  - fine-tuned
+  - rtx4060
+license: apache-2.0
+pipeline_tag: text-generation
+base_model: ynanxiu/olmo3-190M-zh-full
+---
+
+# OLMo3-190M-zh-sft
+
+**187M 参数中文 SFT 对话模型**——让预训练模型学会聊天。
+
+基于 [OLMo3-190M-zh-full](https://huggingface.co/ynanxiu/olmo3-190M-zh-full) 进行有监督微调（SFT），专为中文对话场景训练。
+
+## 📊 模型信息
+
+| 项目 | 详情 |
+|------|------|
+| **基座模型** | [ynanxiu/olmo3-190M-zh-full](https://huggingface.co/ynanxiu/olmo3-190M-zh-full) |
+| **架构** | OLMo3, 12层, d=768, 12头, SwiGLU |
+| **参数量** | 187M |
+| **词表大小** | 48,000 |
+| **训练方式** | SFT（Supervised Fine-Tuning） |
+| **训练数据** | `cmz1024/llm101-olmo3-zh-demo-data` (SFT 对话格式) |
+| **格式** | ChatML (`<|user|>...<|assistant|>...`) |
+
+## 🏋️ 训练详情
+
+| 参数 | 值 |
+|------|-----|
+| 硬件 | NVIDIA RTX 4060 (8GB) |
+| 精度 | bfloat16 |
+| Batch Size | 1 × 32 (gradient_accumulation) |
+| 学习率 | 2e-5 |
+| LR Schedule | cosine + warmup (5%) |
+| Epochs | 2 |
+| 总步数 | 56,470 |
+| 训练耗时 | 31 小时 |
+
+## 📈 训练指标
+
+| 阶段 | Eval Loss | Accuracy | Entropy |
+|------|-----------|----------|---------|
+| 初始 (500步) | 3.36 | 43.1% | 3.388 |
+| 最终 (56,470步) | **2.24** | **54.6%** | **2.572** |
+
+Loss 从 3.36 降至 2.24，下降 33%；accuracy 从 43% 升至 55%。
+
+## 💬 使用方法
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_id = "ynanxiu/olmo3-190M-zh-sft"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    attn_implementation="sdpa",
+    trust_remote_code=True,
+)
+
+# 对话模式（推荐）
+messages = [{"role": "user", "content": "你好，请用简单的语言解释什么是机器学习"}]
+formatted = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True
+)
+inputs = tokenizer(formatted, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=200,
+        temperature=0.3,        # ⚠️ 推荐 0.3，避免重复
+        top_p=0.9,
+        do_sample=True,
+        repetition_penalty=1.15,
+        pad_token_id=tokenizer.pad_token_id,
+    )
+
+response = tokenizer.decode(outputs[0][len(inputs["input_ids"][0]):], skip_special_tokens=True)
+print(response)
+```
+
+### ⚙️ 推荐生成参数
+
+| 参数 | 推荐值 | 说明 |
+|------|--------|------|
+| temperature | **0.3** | 190M 模型不建议超过 0.5，高温导致严重重复 |
+| repetition_penalty | 1.15~1.2 | 抑制重复 |
+| top_p | 0.9 | 标准核采样 |
+| max_new_tokens | 150~200 | 太长容易退化 |
+
+## 🎮 在线体验
+
+👉 **[HuggingFace Space](https://huggingface.co/spaces/ynanxiu/olmo3-190M-zh-sft)** — 直接在浏览器里对话
+
+## 📝 生成效果
+
+### 成功案例 ✅
+
+```
+用户：你好，请问你能帮我做什么？
+助手：当然可以！请告诉我你最近在做什么，或者有什么具体问题
+      我可以帮你解决？我会尽力提供帮助。
+
+用户：今天天气不错，我准备
+助手：我无法提供实时天气信息或天气预报。建议您查看天气应用
+      或网站获取最新天气信息。如果您有其他问题，欢迎随时提问！
+```
+
+### 已知限制 ⚠️
+
+- **重复问题**：190M 参数较小，长回复可能出现重复。建议 temperature ≤ 0.3 + repetition_penalty ≥ 1.15。
+- **知识幻觉**：可能生成不准确的事实（如自称 "minimind"），SFT 数据影响了模型认知。
+- **复杂推理**：190M 参数量不适合复杂多轮推理任务。
+
+## 🔗 相关模型
+
+| 模型 | 类型 | 链接 |
+|------|------|------|
+| OLMo3-190M-zh-full | 基础预训练 | [HF](https://huggingface.co/ynanxiu/olmo3-190M-zh-full) |
+| OLMo3-190M-zh-nano | 小模型持续预训练 | [HF](https://huggingface.co/ynanxiu/olmo3-190M-zh-nano) |
+
+## 📜 License
+
+Apache 2.0
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,6 @@
+{% for message in messages %}{% if message['role'] == 'assistant' %}{% generation %}{{ '<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+' }}{% endgeneration %}{% else %}{{ '<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,44 @@
+{
+  "architectures": [
+    "Olmo3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "dtype": "bfloat16",
+  "eos_token_id": 5,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 4096,
+  "model_type": "olmo3",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.9.0",
+  "use_cache": false,
+  "vocab_size": 48000
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": 0,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 1,
+  "transformers_version": "5.9.0",
+  "use_cache": true
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:964abce0c580c2c1a4a53a665ad390a203dd0009592541c8aae0291c837c2658
+size 374038864
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,16 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|bos|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 4096,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|unk|>"
+}