130 lines
4.9 KiB
Markdown
130 lines
4.9 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
datasets:
|
||
- HuggingFaceFW/fineweb-edu
|
||
- mlfoundations/dclm-baseline-1.0-parquet
|
||
tags:
|
||
- boris
|
||
- opencerebral
|
||
- gpt2
|
||
- 125M
|
||
---
|
||
|
||
|
||
|
||
# Boris-1.3-125M
|
||
|
||
> **Note:** New Millennium Artificial Intelligence (NMAI) has been renamed
|
||
> **OpenCerebral**. The organization, models, and maintainers are unchanged —
|
||
> only the name is new. Older references to NMAI (including the previous
|
||
> `KSP-NMAI` repository paths) refer to OpenCerebral.
|
||
|
||
Boris-1.3-125M is a 125 million-parameter language model created by OpenCerebral.
|
||
It extends the original Boris-125M base checkpoint
|
||
with additional continued pretraining aimed at closing gaps found in Boris-125M's
|
||
own benchmark results (see *Continued pretraining* below).
|
||
|
||
This is a **base (pretrained) model**. It has not been instruction-tuned and does
|
||
not follow instructions or hold a conversation — it continues text. For an
|
||
instruction-following version, see
|
||
[opencerebral/Boris-1.3-125M-Instruct](https://huggingface.co/opencerebral/Boris-1.3-125M-Instruct).
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
tok = AutoTokenizer.from_pretrained("opencerebral/Boris-1.3-125M")
|
||
model = AutoModelForCausalLM.from_pretrained("opencerebral/Boris-1.3-125M")
|
||
|
||
ids = tok("The ocean is", return_tensors="pt").input_ids
|
||
out = model.generate(ids, max_new_tokens=40, do_sample=True, top_p=0.95)
|
||
print(tok.decode(out[0], skip_special_tokens=True))
|
||
```
|
||
|
||
## Details
|
||
|
||
| | |
|
||
|---|---|
|
||
| Architecture | GPT-2 (pre-LN, learned positional embeddings, tied embeddings) |
|
||
| Layers / heads / d_model | 12 / 12 / 768 |
|
||
| Context length | 1024 |
|
||
| Vocab | 50304 (GPT-NeoX-20B BPE, padded) |
|
||
| Tokenizer | `EleutherAI/gpt-neox-20b` |
|
||
| Precision | trained in bf16 autocast with fp32 master weights |
|
||
|
||
## Base model training
|
||
|
||
The Boris-125M base checkpoint was trained on 2.50B tokens of FineWeb-Edu for
|
||
33h 38m 48s on one RTX 3060.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Final loss | 3.2998 |
|
||
| Final grad norm | 0.281 |
|
||
| Final learning rate | 6.00e-05 |
|
||
|
||
## Continued pretraining
|
||
|
||
Boris-125M's benchmark results showed the same FineWeb-Edu-driven gap seen at
|
||
75M. Boris-1.3-125M adds seven sequential continued-pretraining passes on top
|
||
of the base checkpoint, each with a re-warmed learning rate, extending total
|
||
training by roughly 2.66B tokens (~106% more than the original 2.50B-token
|
||
pretraining run):
|
||
|
||
| Pass | Data | Tokens | Wall-clock (RTX 3060) |
|
||
|---|---|---|---|
|
||
| 1 | FineWeb-Edu | 0.6B | ~6.9h *(estimated)* |
|
||
| 2 | DCLM-baseline | 1.0B | ~11.9h *(estimated)* |
|
||
| 3 | FineWeb-Edu | 0.1B | ~1.2h *(estimated)* |
|
||
| 4 | FineWeb-Edu | 0.1B | ~1.2h *(estimated)* |
|
||
| 5 | FineWeb-Edu | 0.1B | ~1.2h *(estimated)* |
|
||
| 6 | FineWeb-Edu-leaning | 0.91B | ~7.8h+ *(required a restart)* |
|
||
| 7 | DCLM-baseline | 0.6B | ~7.1h *(estimated)* |
|
||
|
||
Final training loss, grad norm, and learning rate for pass 7 were not
|
||
preserved and are not available for this card.
|
||
|
||
**Why this recipe:** DCLM improves fluency/coherence tasks (LAMBADA, WinoGrande)
|
||
but tends to cost ARC-Easy/ARC-Challenge performance. Unlike Boris-1.3-75M,
|
||
this run leads with FineWeb-Edu before DCLM specifically to test whether that
|
||
order avoids the ARC regression — it did. The three small FineWeb-Edu passes
|
||
(3–5) and the larger pass 6 were run to test how far ARC-Challenge and
|
||
mean score could be pushed with small, individually-measured increments.
|
||
|
||
| Task | Boris-125M | +FineWeb-Edu | +DCLM | +FineWeb-Edu ×3 | +FineWeb-Edu | Boris-1.3-125M |
|
||
|---|---|---|---|---|---|---|
|
||
| HellaSwag (acc_norm) | 29.33 | 29.40 | 29.24 | 29.50 | 29.79 | 29.68 |
|
||
| PIQA (acc_norm) | 59.74 | 60.72 | 60.72 | 61.43 | 60.61 | 61.32 |
|
||
| WinoGrande (acc) | 49.72 | 50.36 | 50.59 | 50.28 | 51.70 | 52.72 |
|
||
| ARC-Easy (acc_norm) | 41.75 | 41.41 | 41.54 | 41.79 | 43.01 | 42.51 |
|
||
| ARC-Challenge (acc_norm) | 23.89 | 24.74 | 23.72 | 24.40 | 25.09 | 24.23 |
|
||
| LAMBADA (acc) | 22.86 | 23.17 | 24.63 | 24.74 | 23.23 | 25.79 |
|
||
| **Mean-6** | **37.88** | **38.30** | **38.41** | **38.69** | **38.91** | **39.38** |
|
||
|
||
|
||
|
||
## Limitations
|
||
|
||
A base model of this size will produce text that is frequently inaccurate,
|
||
inconsistent, or offensive. It has received no alignment or safety tuning and
|
||
should not be used for factual reference or deployed without supervision.
|
||
|
||
## Copyright & License
|
||
|
||
*Copyright 2026 Joseph Jones*
|
||
|
||
This project and all associated files (the "Work") are licensed under the Apache
|
||
License, Version 2.0 (the "License"); you may not use this project except in
|
||
compliance with the License. You may obtain a copy of the License at:
|
||
|
||
http://www.apache.org/licenses/LICENSE-2.0
|
||
|
||
Unless required by applicable law or agreed to in writing, software distributed
|
||
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
|
||
CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||
specific language governing permissions and limitations under the License. |