初始化项目，由ModelHub XC社区提供模型

Model: opencerebral/Boris-1.3-125M Source: Original Platform
2026-09-15 16:51:17 +08:00
commit c0597a7eba
9 changed files with 250793 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,39 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+Boris-75M.png filter=lfs diff=lfs merge=lfs -text
+Boris-1.3-75M.png filter=lfs diff=lfs merge=lfs -text
+benchmarks.png filter=lfs diff=lfs merge=lfs -text
+Boris-1.3-125M.png filter=lfs diff=lfs merge=lfs -text
--- a/Boris-1.3-125M.png
+++ b/Boris-1.3-125M.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:988da15085a90af35ca7ba6cc5a1ed84edf807d2f29eb833cecf93c09e5d2815
+size 788194
--- a/README.md
+++ b/README.md
@@ -0,0 +1,130 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+- HuggingFaceFW/fineweb-edu
+- mlfoundations/dclm-baseline-1.0-parquet
+tags:
+- boris
+- opencerebral
+- gpt2
+- 125M
+---
+
+![Boris](Boris-1.3-125M.png)
+
+# Boris-1.3-125M
+
+> **Note:** New Millennium Artificial Intelligence (NMAI) has been renamed
+> **OpenCerebral**. The organization, models, and maintainers are unchanged —
+> only the name is new. Older references to NMAI (including the previous
+> `KSP-NMAI` repository paths) refer to OpenCerebral.
+
+Boris-1.3-125M is a 125 million-parameter language model created by OpenCerebral.
+It extends the original Boris-125M base checkpoint
+with additional continued pretraining aimed at closing gaps found in Boris-125M's
+own benchmark results (see *Continued pretraining* below).
+
+This is a **base (pretrained) model**. It has not been instruction-tuned and does
+not follow instructions or hold a conversation — it continues text. For an
+instruction-following version, see
+[opencerebral/Boris-1.3-125M-Instruct](https://huggingface.co/opencerebral/Boris-1.3-125M-Instruct).
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("opencerebral/Boris-1.3-125M")
+model = AutoModelForCausalLM.from_pretrained("opencerebral/Boris-1.3-125M")
+
+ids = tok("The ocean is", return_tensors="pt").input_ids
+out = model.generate(ids, max_new_tokens=40, do_sample=True, top_p=0.95)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+## Details
+
+| | |
+|---|---|
+| Architecture | GPT-2 (pre-LN, learned positional embeddings, tied embeddings) |
+| Layers / heads / d_model | 12 / 12 / 768 |
+| Context length | 1024 |
+| Vocab | 50304 (GPT-NeoX-20B BPE, padded) |
+| Tokenizer | `EleutherAI/gpt-neox-20b` |
+| Precision | trained in bf16 autocast with fp32 master weights |
+
+## Base model training
+
+The Boris-125M base checkpoint was trained on 2.50B tokens of FineWeb-Edu for
+33h 38m 48s on one RTX 3060.
+
+| | |
+|---|---|
+| Final loss | 3.2998 |
+| Final grad norm | 0.281 |
+| Final learning rate | 6.00e-05 |
+
+## Continued pretraining
+
+Boris-125M's benchmark results showed the same FineWeb-Edu-driven gap seen at
+75M. Boris-1.3-125M adds seven sequential continued-pretraining passes on top
+of the base checkpoint, each with a re-warmed learning rate, extending total
+training by roughly 2.66B tokens (~106% more than the original 2.50B-token
+pretraining run):
+
+| Pass | Data | Tokens | Wall-clock (RTX 3060) |
+|---|---|---|---|
+| 1 | FineWeb-Edu | 0.6B | ~6.9h *(estimated)* |
+| 2 | DCLM-baseline | 1.0B | ~11.9h *(estimated)* |
+| 3 | FineWeb-Edu | 0.1B | ~1.2h *(estimated)* |
+| 4 | FineWeb-Edu | 0.1B | ~1.2h *(estimated)* |
+| 5 | FineWeb-Edu | 0.1B | ~1.2h *(estimated)* |
+| 6 | FineWeb-Edu-leaning | 0.91B | ~7.8h+ *(required a restart)* |
+| 7 | DCLM-baseline | 0.6B | ~7.1h *(estimated)* |
+
+Final training loss, grad norm, and learning rate for pass 7 were not
+preserved and are not available for this card.
+
+**Why this recipe:** DCLM improves fluency/coherence tasks (LAMBADA, WinoGrande)
+but tends to cost ARC-Easy/ARC-Challenge performance. Unlike Boris-1.3-75M,
+this run leads with FineWeb-Edu before DCLM specifically to test whether that
+order avoids the ARC regression — it did. The three small FineWeb-Edu passes
+(3–5) and the larger pass 6 were run to test how far ARC-Challenge and
+mean score could be pushed with small, individually-measured increments.
+
+| Task | Boris-125M | +FineWeb-Edu | +DCLM | +FineWeb-Edu ×3 | +FineWeb-Edu | Boris-1.3-125M |
+|---|---|---|---|---|---|---|
+| HellaSwag (acc_norm) | 29.33 | 29.40 | 29.24 | 29.50 | 29.79 | 29.68 |
+| PIQA (acc_norm) | 59.74 | 60.72 | 60.72 | 61.43 | 60.61 | 61.32 |
+| WinoGrande (acc) | 49.72 | 50.36 | 50.59 | 50.28 | 51.70 | 52.72 |
+| ARC-Easy (acc_norm) | 41.75 | 41.41 | 41.54 | 41.79 | 43.01 | 42.51 |
+| ARC-Challenge (acc_norm) | 23.89 | 24.74 | 23.72 | 24.40 | 25.09 | 24.23 |
+| LAMBADA (acc) | 22.86 | 23.17 | 24.63 | 24.74 | 23.23 | 25.79 |
+| **Mean-6** | **37.88** | **38.30** | **38.41** | **38.69** | **38.91** | **39.38** |
+
+![Benchmarks](benchmarks.png)
+
+## Limitations
+
+A base model of this size will produce text that is frequently inaccurate,
+inconsistent, or offensive. It has received no alignment or safety tuning and
+should not be used for factual reference or deployed without supervision.
+
+## Copyright & License
+
+*Copyright 2026 Joseph Jones*
+
+This project and all associated files (the "Work") are licensed under the Apache
+License, Version 2.0 (the "License"); you may not use this project except in
+compliance with the License. You may obtain a copy of the License at:
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed
+under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
--- a/benchmarks.png
+++ b/benchmarks.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a6f0d0a247172581ee5215560c77775d296701c32e552208ac2fcfaab8ab957
+size 181002
--- a/config.json
+++ b/config.json
@@ -0,0 +1,35 @@
+{
+  "activation_function": "gelu",
+  "add_cross_attention": false,
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.0,
+  "bos_token_id": 0,
+  "dtype": "float32",
+  "embd_pdrop": 0.0,
+  "eos_token_id": 0,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "pad_token_id": null,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.0,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.13.1",
+  "use_cache": true,
+  "vocab_size": 50304
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.13.1",
+  "use_cache": true
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b4968af99656b251341a9a6c50d242a3d63d2670d93f41d29b653b0575aa6993
+size 497918592
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,14 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|padding|>",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "trim_offsets": true,
+  "unk_token": "<|endoftext|>"
+}