初始化项目，由ModelHub XC社区提供模型

Model: Fengshenbang/Ziya2-13B-Base
Source: Original Platform

.gitattributes

@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bin.* filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*.tfevents* filter=lfs diff=lfs merge=lfs -text
+*.db* filter=lfs diff=lfs merge=lfs -text
+*.ark* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,129 @@
+---
+tasks:
+
+- text-generation
+
+model_type:
+
+- gpt
+- llama
+
+domain:
+
+- nlp
+
+license: Apache License 2.0
+
+language:
+
+- en
+- zh
+- cn
+
+tags:
+- transformer
+- 封神榜
+---
+# Ziya2-13B-Base
+
+- Main Page:[Fengshenbang](https://fengshenbang-lm.com/)
+- Github: [Fengshenbang-LM](https://github.com/IDEA-CCNL/Fengshenbang-LM)
+
+
+# 姜子牙系列模型
+
+- [Ziya-LLaMA-13B-v1](https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-13B-v1)
+- [Ziya-LLaMA-7B-Reward](https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-7B-Reward)
+- [Ziya-LLaMA-13B-Pretrain-v1](https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-13B-Pretrain-v1)
+- [Ziya-BLIP2-14B-Visual-v1](https://huggingface.co/IDEA-CCNL/Ziya-BLIP2-14B-Visual-v1)
+
+## 简介 Brief Introduction
+
+Ziya2-13B-Base 是基于LLaMa2的130亿参数大规模预训练模型，针对中文分词优化，并完成了中英文 650B tokens 的增量预训练，进一步提升了中文生成和理解能力。
+
+The Ziya2-13B-Base is a large-scale pre-trained model based on LLaMA2 with 13 billion parameters. We optimizes LLaMAtokenizer on chinese, and incrementally train 650 billion tokens of data based on LLaMa2-13B model, which significantly improved the understanding and generation ability on Chinese. 
+![联合宣传](./img1.png)
+## 模型分类 Model Taxonomy
+
+| 需求 Demand  | 任务 Task | 系列 Series | 模型 Model | 参数 Parameter | 额外 Extra |
+|:----------:|:-------:|:---------:|:--------:|:------------:|:---------------:|
+| 通用 General | AGI模型   | 姜子牙 Ziya  | LLaMA2   | 13B          | English&Chinese |
+
+## 模型信息 Model Information
+
+### 继续预训练 Continual Pretraining
+
+Meta在2023年7月份发布了Llama2系列大模型，相比于LLaMA1的1.4万亿Token数据，Llama2预训练的Token达到了2万亿，并在各个榜单中明显超过LLaMA1。
+
+Meta released the Llama2 series of large models in July 2023, with pre-trained tokens reaching 200 billion compared to Llama1's 140 billion tokens, significantly outperforming Llama1 in various rankings.
+
+Ziya2-13B-Base沿用了Ziya-LLaMA-13B高效的中文编解码方式，但采取了更优化的初始化算法使得初始训练loss更低。同时，我们对Fengshen-PT继续训练框架进行了优化，效率方面，整合了FlashAttention2、Apex RMS norm等技术来帮助提升效率，对比Ziya-LLaMA-13B训练速度提升38%(163 TFLOPS/per gpu/per sec)。稳定性方面，我们采取BF16进行训练，修复了底层分布式框架的bug，确保模型能够持续稳定训练，解决了Ziya-LLaMA-13B遇到的训练后期不稳定的问题，并在7.25号进行了直播，最终完成了全部数据的继续训练。我们也发现，模型效果还有进一步提升的趋势，后续也会对Ziya2-13B-Base进行继续优化。
+
+Ziya2-13B-Base retained the efficient Chinese encoding and decoding techniques of Ziya-LLaMA-13B, but employed a more optimized initialization algorithm to achieve lower initial training loss. Additionally, we optimized the Fengshen-PT fine-tuning framework. In terms of efficiency, we integrated technologies such as FlashAttention2 and Apex RMS norm to boost efficiency, resulting in a 38% increase in training speed compared to Ziya-LLaMA-13B (163 TFLOPS per GPU per second). For stability, we used BF16 for training, fixed underlying distributed framework bugs to ensure consistent model training, and resolved the late-stage instability issues encountered in the training of Ziya-LLaMA-13B. We also conducted a live broadcast on July 25th to complete the continued training of all data. We have observed a trend towards further improvements in model performance and plan to continue optimizing Ziya2-13B-Base in the future.
+
+![loss曲线](./img2.png)
+
+### 效果评估 Performance
+
+Ziya2-13B-Base在Llama2-13B的基础上进行了约650B自建高质量中英文数据集的继续训练，在中文、英文、数学、代码等下游理解任务上相对于Llama2-13B取得了明显的提升，相对Ziya-LLaMA-13B也有明显的提升。
+
+The model Ziya2-13B-Base underwent further training on approximately 650 billion self-collected high-quality Chinese and English datasets, building upon the foundation of Llama2-13B. It achieved significant improvements in downstream comprehension tasks such as Chinese, English, mathematics, and code understanding, surpassing Llama2-13B and showing clear advancements compared to Ziya-LLaMA-13B.
+
+![效果评估](./img3.png)
+
+## 使用 Usage 
+
+加载模型，进行的续写：
+
+Load the model and  predicting：
+
+```python3
+from transformers import AutoTokenizer
+from transformers import LlamaForCausalLM
+import torch
+
+query="问题：我国的三皇五帝分别指的是谁？答案：
+model = LlamaForCausalLM.from_pretrained('IDEA-CCNL/Ziya2-13B-Base', torch_dtype=torch.float16, device_map="auto").eval()
+tokenizer = AutoTokenizer.from_pretrained(ckpt)
+input_ids = tokenizer(query, return_tensors="pt").input_ids.to('cuda:0')
+generate_ids = model.generate(
+            input_ids,
+            max_new_tokens=512, 
+            do_sample = True, 
+            top_p = 0.9)
+output = tokenizer.batch_decode(generate_ids)[0]
+print(output)
+```
+
+上面是简单的续写示例，其他更多prompt和玩法，感兴趣的朋友可以下载下来自行发掘。
+
+The above is a simple example of continuing writing. For more prompts and creative ways to use the model, interested individuals can download it and explore further on their own.
+
+## 引用 Citation
+
+如果您在您的工作中使用了我们的模型，可以引用我们的[论文](https://arxiv.org/abs/2311.03301)：
+
+If you are using the resource for your work, please cite the our [paper](https://arxiv.org/abs/2311.03301):
+
+```text
+@article{Ziya2,
+  author    = {Ruyi Gan, Ziwei Wu, Renliang Sun, Junyu Lu, Xiaojun Wu, Dixiang Zhang, Kunhao Pan, Ping Yang, Qi Yang, Jiaxing Zhang, Yan Song},
+  title     = {Ziya2: Data-centric Learning is All LLMs Need},
+  year      = {2023}
+}
+```
+
+You can also cite our [website](https://github.com/IDEA-CCNL/Fengshenbang-LM/):
+
+欢迎引用我们的[网站](https://github.com/IDEA-CCNL/Fengshenbang-LM/):
+
+```text
+@misc{gan2023ziya2,
+      title={Ziya2: Data-centric Learning is All LLMs Need}, 
+      author={Ruyi Gan and Ziwei Wu and Renliang Sun and Junyu Lu and Xiaojun Wu and Dixiang Zhang and Kunhao Pan and Ping Yang and Qi Yang and Jiaxing Zhang and Yan Song},
+      year={2023},
+      eprint={2311.03301},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```

added_tokens.json

@@ -0,0 +1,4 @@
+{
+  "<bot>": 39409,
+  "<human>": 39408
+}

config.json

@@ -0,0 +1,26 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 13824,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 4096,
+  "model_type": "llama",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 40,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-06,
+  "rotary_emb_base": 10000,
+  "rotary_pct": 1,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.30.2",
+  "use_cache": true,
+  "use_parallel_residual": false,
+  "vocab_size": 39424
+}

configuration.json

@@ -0,0 +1 @@
+{"framework":"Pytorch","task":"fill-mask"}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "transformers_version": "4.30.2"
+}

pytorch_model-00001-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58fcec6cc8f64994950f8d609154e9716bae9ffce7718aabafc95a981267284b
+size 896534991

pytorch_model-00002-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f434eb99fbbf675dac0b896853c3c76ac499a9e950adcb8894295f6a1ae28d79
+size 985707823

pytorch_model-00003-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b40a90ada8790140c48e238594eaf3307a510b8d18598722ef1e80aa314268f
+size 917528001

pytorch_model-00004-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cab0c80c7adecd376eca6253ff84dbdee004240302c386072336d6d807e00afa
+size 985707823

pytorch_model-00005-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd4918bffe1c2a9136e5989b44a1c5c5d90f38cde66734f761307a2115742890
+size 917528001

pytorch_model-00006-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3411357e58de4193fa8c75b21929ad7f5f00aca8e626fb8af81d3312480572f
+size 985707823

pytorch_model-00007-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c569f55b71ff92e40c04a8e420377587afdd14b75ee37714c85aed66be81db70
+size 917528001

pytorch_model-00008-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50d38b8b8c8e3380be78e956ecc9873a268c49ccaa8952dc81eb46f0091f2200
+size 985707823

pytorch_model-00009-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c64fe112d9bf0aca8730337ee7651145842cbfe4a171af3ad080834bbd3f5f01
+size 917528001

pytorch_model-00010-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37bbe56a0a3dfc536c106f4d6784410ea2e3065f9f8b9986b2a193623d4ae5f2
+size 985707823

pytorch_model-00011-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0ce324eb37fa8a5987a3aef37c824e97e6e1931ae173382432ca1e84287c94c
+size 917528001

pytorch_model-00012-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd90c2387a7185f5f95e0eabcc924bb4ee202df41af87f98395b75f0e1e1417c
+size 985707823

pytorch_model-00013-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b0ec5133336611bb8fe99ef89218e7e9309b3ab6f9feef1a591aa38e63ec087
+size 917528001

pytorch_model-00014-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc616f57f410066f5e58e941be2bf4ca29e191bb8aa231dd0c9435181195fa87
+size 985707823

pytorch_model-00015-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:edcb8c50e1397947fb55d0c78d3c36c54ab798705ef94cd25205307aff20a45f
+size 917528001

pytorch_model-00016-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93f25f171e62c82808ea32f09e6fc4ab3b3723418bc121e86885f1ef3e38e038
+size 985707823

pytorch_model-00017-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd0cdfff2019e214f12dd7602ff2e9f669d63bc4bfed651d8ffa6baf2482278c
+size 917528001

pytorch_model-00018-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2d36fb07d744bc97a55443fab68fbd9bd776245341d996a5c3848d0a79a9103
+size 985707823

pytorch_model-00019-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed579c633f7f4f51b9d14952e5b8417bef607710dfd28cd06d80fefbc570feed
+size 917528001

pytorch_model-00020-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3caef721b8b15706a4a566a108dbac4f28c8d1f3d35456ba2f8fab6b1275d3d0
+size 985707823

pytorch_model-00021-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b98efcf6b3a72acb7cb1ea62ab519304e3d18481f54117e961aa7d28af9aea0f
+size 917528001

pytorch_model-00022-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:394ac6396848038f4591343393edd996d8c3f206d35f5a1135021466e0f52cf8
+size 985707823

pytorch_model-00023-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4785f54e41f1b9c2a098061b70185e3a01f4467b37fcf88b0f2fdffe965f83c
+size 917528001

pytorch_model-00024-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71ae6d94acb1b488b534cc2bcd0ab147c625e5f1d0b38dddae47f7afe970abd2
+size 985707823

pytorch_model-00025-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05cb90f9fafb73b094a7b91d561d7aec560f2e6164a19bd0081942d60cc27bf4
+size 917528001

pytorch_model-00026-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43ed8f14508cddcf3e1e689434a245ebc333f7da24bce346ad41105d39642c76
+size 985707823

pytorch_model-00027-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:523a61944cf3416497fd41bca283ea7e592e6b2a76c147a84f2ae6ef04d3323c
+size 917528001

pytorch_model-00028-of-00028.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb4302a156960a3c3d76efd371812ae470edde8c1eaa8ef45925565ca6784b64
+size 545291867

pytorch_model.bin.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd2aeaa1baf2bcc094dfe61143c5f49147d8ba1cba70d1182d9f4bd732caf86c
+size 33444

special_tokens_map.json

@@ -0,0 +1,39 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<human>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<bot>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fd7e445833dd0889206aba242c2a51ecbae2437fd328d1759a35475fd8c0423
+size 588619

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}