初始化项目，由ModelHub XC社区提供模型

Model: Duxiaoman-DI/XuanYuan-6B Source: Original Platform
2026-05-19 20:03:57 +08:00
commit f732c75832
9 changed files with 102079 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bin.* filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*.tfevents* filter=lfs diff=lfs merge=lfs -text
+*.db* filter=lfs diff=lfs merge=lfs -text
+*.ark* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,65 @@
+---
+frameworks:
+- Pytorch
+license: Apache License 2.0
+tasks:
+- text-generation
+---
+
+## 介绍
+
+XuanYuan-6B系列模型是采用类LLaMA架构，从零开始进行预训练的金融大模型。我们构建了大规模、多样化、高质量的训练语料对模型进行了充分预训练，使模型具备各项能力。此外我们构建了丰富、高质量的问答数据和人类偏好数据，并通过指令微调和强化学习进一步对齐模型表现和人类偏好，显著提升了模型在对话场景中的表现。各项评估显示，XuanYuan-6B不仅具备较强的通用能力，更具备强大的金融能力。更多细节请参考我们的技术报告：[Report](https://github.com/Duxiaoman-DI/XuanYuan/blob/main/xuanyuan_6b_report.md)
+
+XuanYuan-6B系列模型包含基座模型XuanYuan-6B，经指令微调和强化对齐的chat模型XuanYuan-6B-Chat，以及chat模型的量化版本XuanYuan-6B-Chat-4bit和XuanYuan-6B-Chat-8bit。各个模型的链接为：
+
+| 基座模型                                                     | Chat模型                                                     | 8-bit量化Chat模型                                            | 4-bit量化Chat模型                                          |
+| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
+| [XuanYuan-6B](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-6B) | [XuanYuan-6B-Chat](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-6B-Chat) | [XuanYuan-6B-Chat-8bit](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-6B-Chat-8bit ) |  [XuanYuan-6B-Chat-4bit](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-6B-Chat-4bit) |
+
+
+主要特点：
+
+* 收集多个领域大量的训练预料，进行了多维度数据清洗和去重，保证数据的量级和质量
+* 从零开始预训练，预训练中动态调整数据配比，模型基座能力较强
+* 结合Self-QA方法构建高质量问答数据，采用混合训练方式进行监督微调
+* 构建高质量人类偏好数据训练奖励模型并进行强化训练，对齐模型表现和人类偏好
+* 模型尺寸小并包含量化版本，硬件要求低，适用性更强
+* 在多个榜单和人工评估中均展现出良好的性能，具备领先的金融能力
+
+## 模型细节
+
+XuanYuan-6B具有4096个隐藏单元，由30层和32个注意⼒头组成。为了融⼊位置信息，我们采⽤了RoPE作为位置嵌⼊技术。模型中使⽤的激活函数是SwiGLU，并使⽤RMSNorm进⾏归⼀化处理。在训练过程中，我们将最⼤序列⻓度设置为2048个token。词表的⼤⼩为39438，与我们先前模型（XuanYuan-13B、XuanYuan-70B）使⽤的词表⼀致。
+
+## 训练细节
+
+训练前，我们从不同领域收集了大量训练语料，并对数据进行一系列处理来提升质量。
+
+预训练中，我们不断评估模型在特定任务或基准上的性能，并根据评估结果动态调整不同来源的训练数据配⽐，不断优化模型训练过程，提升模型各项能力。
+
+我们利用Self-QA的方法构建了高质量指令微调数据集，并结合无监督语言模型任务对预训练后的模型进行了混合微调。在增强模型chat场景下各项能力的同时，保证其泛化性。
+
+最后，我们通过人工标注的方式构建了高质量的偏好数据，由此训练奖励模型并进行强化对齐训练，使其表现对齐人类偏好，以继续提升模型各项能力。
+
+## 使用方法
+XuanYuan-6B基座模型、chat模型及其量化模型的使用方法和[XuanYuan-70B](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-70B/summary)，[XuanYuan2-70B](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan2-70B/summary)类似，但是tokenizer加载方式和在对话场景中使用的prompt格式不同（不包含system message）。下面以XuanYuan-6B-Chat模型为例，来展示XuanYuan-6B系列模型的使用方法。
+```python
+import torch
+from transformers import LlamaForCausalLM, AutoTokenizer
+
+model_name_or_path = "your/model/path/"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
+model = LlamaForCausalLM.from_pretrained(model_name_or_path, device_map="auto")
+model.eval()
+
+seps = [" ", "</s>"]
+roles = ["Human", "Assistant"]
+
+content = "介绍下你自己"
+prompt = seps[0] + roles[0] + ": " + content + seps[0] + roles[1] + ":"
+print(f"输入: {content}")
+inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.95)
+outputs = tokenizer.decode(outputs.cpu()[0][len(inputs.input_ids[0]):], skip_special_tokens=True)
+print(f"输出: {outputs}")
+```
--- a/config.json
+++ b/config.json
@@ -0,0 +1,27 @@
+{
+  "_name_or_path": "xuanyuan_6b",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 8192,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 30,
+  "num_key_value_heads": 32,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.33.1",
+  "use_cache": true,
+  "vocab_size": 39438
+}
--- a/configuration.json
+++ b/configuration.json
@@ -0,0 +1 @@
+{"framework":"Pytorch","task":"text-generation"}
--- a/pytorch_model.bin
+++ b/pytorch_model.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2b147d331faaabcaf0470ef2e8c64d4b31112525231265eb25bb41ca3d532a5
+size 12789222574
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,6 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "pad_token": "</s>",
+  "unk_token": "<unk>"
+}
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer.model
+++ b/tokenizer.model
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21c5025309cbeced41b89d100d24b0409758d39604bd2d4df63c7b8f85f6d7f3
+size 592204
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,35 @@
+{
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "fast_tokenizer": true,
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "use_default_system_prompt": true
+}
				`@@ -0,0 +1 @@`
				`{"framework":"Pytorch","task":"text-generation"}`