初始化项目，由ModelHub XC社区提供模型

Model: TheOneWhoWill/Shibai-700M-Base
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,71 @@
+---
+license: apache-2.0
+datasets:
+- HuggingFaceFW/fineweb
+- HuggingFaceFW/fineweb-edu
+- bigcode/the-stack
+- HuggingFaceFW/finewiki
+- TheOneWhoWill/naruto-boruto-wiki
+- CausalLM/Refined-Anime-Text
+- RyokoAI/Fandom23K
+- kurumikz/animepedia
+- openai/gsm8k
+- HuggingFaceH4/MATH-500
+- TIGER-Lab/MMLU-Pro
+- tatsu-lab/alpaca
+- open-r1/OpenR1-Math-220k
+- TIGER-Lab/MathInstruct
+- meta-math/MetaMathQA
+- open-web-math/open-web-math
+- hkust-nlp/dart-math-hard
+- EleutherAI/hendrycks_math
+language:
+- en
+library_name: transformers
+---
+# Shibai 700m Base
+
+This is a 700m parameter LLaMA based model pre-trained with over 18 billion tokens of English text, Math, and Python code at a 2k native context length. It was trained in two stages, first to establish a strong foundational model that can comprehend and generate text, and then to improve its reasoning and coding capabilities with a continued pre-training stage. The use of RoPE scaling allows the model to extrapolate to longer context lengths with the same quality, up to 8k tokens, without any additional training. Although this model may be considered under-trained compared to models such as Qwen 3 0.6B or Qwen 3.5 0.8B, it is still capable of generating coherent and contextually relevant text across a wide range of topics. It should also be noted that this model was trained for 400 hours exclusively on a single RTX 5070 Ti GPU, which is a significant achievement in terms of resource efficiency.
+
+## Model Specs Overview
+* Number of parameters: 678.4M
+* Hidden size: 1280
+* Number of layers: 28
+* Number of attention heads: 16 (KV: 4)
+* Intermediate size: 4480
+* Sequence length: 2048
+* RoPE Theta: 10,000
+* Vocab size: 32,000
+
+## Getting Started
+To use this model, you can load it with the Hugging Face Transformers library. To load the model you can use the following code snippet:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+MODEL_ID = "TheOneWhoWill/Shibai-700M-Base"
+
+processor = AutoTokenizer.from_pretrained(MODEL_ID)
+model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto")
+```
+
+Then once the model is in memory you can start generating output with the following code snippet:
+
+```python
+prompt = "Once upon a time"
+inputs = processor(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(
+	**inputs,
+	max_new_tokens=256,
+	do_sample=True,
+	temperature=0.3,
+	top_p=0.95,
+	top_k=50,
+	repetition_penalty=1.15,
+	no_repeat_ngram_size=3,
+)
+generated_text = processor.decode(outputs[0], skip_special_tokens=True)
+print(generated_text)
+```
+
+Keep in mind that this model was pre-trained at a 2k context length and so you can't expect it to generate a stop token. This model works well for simple next-token prediction tasks but isn't fine tuned for instruction following or chat tasks. For those tasks I am planning on releasing a fine-tuned version of this model in the near future.

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": 2,
+  "head_dim": 80,
+  "hidden_act": "silu",
+  "hidden_size": 1280,
+  "initializer_range": 0.02,
+  "intermediate_size": 4480,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 4,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.5.0",
+  "use_cache": false,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.5.0",
+  "use_cache": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:521a7170da81d7d1aa7ffb9398c1f78afb6a0732ccd527be9920b8122c6b820c
+size 1356770288

tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<eos>",
+  "is_local": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}