初始化项目，由ModelHub XC社区提供模型
Model: opencerebral/Boris-1.3-125M Source: Original Platform
This commit is contained in:
ModelHub XC
130
README.md
Normal file
130
README.md
Normal file
@@ -0,0 +1,130 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
datasets:
|
||||
- HuggingFaceFW/fineweb-edu
|
||||
- mlfoundations/dclm-baseline-1.0-parquet
|
||||
tags:
|
||||
- boris
|
||||
- opencerebral
|
||||
- gpt2
|
||||
- 125M
|
||||
---
|
||||
|
||||
|
||||
|
||||
# Boris-1.3-125M
|
||||
|
||||
> **Note:** New Millennium Artificial Intelligence (NMAI) has been renamed
|
||||
> **OpenCerebral**. The organization, models, and maintainers are unchanged —
|
||||
> only the name is new. Older references to NMAI (including the previous
|
||||
> `KSP-NMAI` repository paths) refer to OpenCerebral.
|
||||
|
||||
Boris-1.3-125M is a 125 million-parameter language model created by OpenCerebral.
|
||||
It extends the original Boris-125M base checkpoint
|
||||
with additional continued pretraining aimed at closing gaps found in Boris-125M's
|
||||
own benchmark results (see *Continued pretraining* below).
|
||||
|
||||
This is a **base (pretrained) model**. It has not been instruction-tuned and does
|
||||
not follow instructions or hold a conversation — it continues text. For an
|
||||
instruction-following version, see
|
||||
[opencerebral/Boris-1.3-125M-Instruct](https://huggingface.co/opencerebral/Boris-1.3-125M-Instruct).
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
tok = AutoTokenizer.from_pretrained("opencerebral/Boris-1.3-125M")
|
||||
model = AutoModelForCausalLM.from_pretrained("opencerebral/Boris-1.3-125M")
|
||||
|
||||
ids = tok("The ocean is", return_tensors="pt").input_ids
|
||||
out = model.generate(ids, max_new_tokens=40, do_sample=True, top_p=0.95)
|
||||
print(tok.decode(out[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## Details
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Architecture | GPT-2 (pre-LN, learned positional embeddings, tied embeddings) |
|
||||
| Layers / heads / d_model | 12 / 12 / 768 |
|
||||
| Context length | 1024 |
|
||||
| Vocab | 50304 (GPT-NeoX-20B BPE, padded) |
|
||||
| Tokenizer | `EleutherAI/gpt-neox-20b` |
|
||||
| Precision | trained in bf16 autocast with fp32 master weights |
|
||||
|
||||
## Base model training
|
||||
|
||||
The Boris-125M base checkpoint was trained on 2.50B tokens of FineWeb-Edu for
|
||||
33h 38m 48s on one RTX 3060.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Final loss | 3.2998 |
|
||||
| Final grad norm | 0.281 |
|
||||
| Final learning rate | 6.00e-05 |
|
||||
|
||||
## Continued pretraining
|
||||
|
||||
Boris-125M's benchmark results showed the same FineWeb-Edu-driven gap seen at
|
||||
75M. Boris-1.3-125M adds seven sequential continued-pretraining passes on top
|
||||
of the base checkpoint, each with a re-warmed learning rate, extending total
|
||||
training by roughly 2.66B tokens (~106% more than the original 2.50B-token
|
||||
pretraining run):
|
||||
|
||||
| Pass | Data | Tokens | Wall-clock (RTX 3060) |
|
||||
|---|---|---|---|
|
||||
| 1 | FineWeb-Edu | 0.6B | ~6.9h *(estimated)* |
|
||||
| 2 | DCLM-baseline | 1.0B | ~11.9h *(estimated)* |
|
||||
| 3 | FineWeb-Edu | 0.1B | ~1.2h *(estimated)* |
|
||||
| 4 | FineWeb-Edu | 0.1B | ~1.2h *(estimated)* |
|
||||
| 5 | FineWeb-Edu | 0.1B | ~1.2h *(estimated)* |
|
||||
| 6 | FineWeb-Edu-leaning | 0.91B | ~7.8h+ *(required a restart)* |
|
||||
| 7 | DCLM-baseline | 0.6B | ~7.1h *(estimated)* |
|
||||
|
||||
Final training loss, grad norm, and learning rate for pass 7 were not
|
||||
preserved and are not available for this card.
|
||||
|
||||
**Why this recipe:** DCLM improves fluency/coherence tasks (LAMBADA, WinoGrande)
|
||||
but tends to cost ARC-Easy/ARC-Challenge performance. Unlike Boris-1.3-75M,
|
||||
this run leads with FineWeb-Edu before DCLM specifically to test whether that
|
||||
order avoids the ARC regression — it did. The three small FineWeb-Edu passes
|
||||
(3–5) and the larger pass 6 were run to test how far ARC-Challenge and
|
||||
mean score could be pushed with small, individually-measured increments.
|
||||
|
||||
| Task | Boris-125M | +FineWeb-Edu | +DCLM | +FineWeb-Edu ×3 | +FineWeb-Edu | Boris-1.3-125M |
|
||||
|---|---|---|---|---|---|---|
|
||||
| HellaSwag (acc_norm) | 29.33 | 29.40 | 29.24 | 29.50 | 29.79 | 29.68 |
|
||||
| PIQA (acc_norm) | 59.74 | 60.72 | 60.72 | 61.43 | 60.61 | 61.32 |
|
||||
| WinoGrande (acc) | 49.72 | 50.36 | 50.59 | 50.28 | 51.70 | 52.72 |
|
||||
| ARC-Easy (acc_norm) | 41.75 | 41.41 | 41.54 | 41.79 | 43.01 | 42.51 |
|
||||
| ARC-Challenge (acc_norm) | 23.89 | 24.74 | 23.72 | 24.40 | 25.09 | 24.23 |
|
||||
| LAMBADA (acc) | 22.86 | 23.17 | 24.63 | 24.74 | 23.23 | 25.79 |
|
||||
| **Mean-6** | **37.88** | **38.30** | **38.41** | **38.69** | **38.91** | **39.38** |
|
||||
|
||||
|
||||
|
||||
## Limitations
|
||||
|
||||
A base model of this size will produce text that is frequently inaccurate,
|
||||
inconsistent, or offensive. It has received no alignment or safety tuning and
|
||||
should not be used for factual reference or deployed without supervision.
|
||||
|
||||
## Copyright & License
|
||||
|
||||
*Copyright 2026 Joseph Jones*
|
||||
|
||||
This project and all associated files (the "Work") are licensed under the Apache
|
||||
License, Version 2.0 (the "License"); you may not use this project except in
|
||||
compliance with the License. You may obtain a copy of the License at:
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed
|
||||
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
|
||||
CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
Reference in New Issue
Block a user