83 lines
3.4 KiB
Markdown
83 lines
3.4 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags:
|
||
|
|
- text-generation
|
||
|
|
- pretrained
|
||
|
|
- base-model
|
||
|
|
- from-scratch
|
||
|
|
- qwen3
|
||
|
|
library_name: transformers
|
||
|
|
---
|
||
|
|
|
||
|
|
# Rivet-1B-PT (base model)
|
||
|
|
|
||
|
|
The **pretrained base** of Rivet-1B: a 1.09B parameter language model trained **entirely from scratch** (custom tokenizer, custom corpus, no inherited weights). This repo holds the raw pretrained checkpoint (v9 lineage), before any supervised fine-tuning.
|
||
|
|
|
||
|
|
> Uses the Qwen3 *architecture* for serving compatibility. Shares **no weights** with Qwen: the tokenizer (custom 32k SentencePiece) and every parameter were trained from zero.
|
||
|
|
|
||
|
|
**Note:** this is the raw base model. It completes text; it does not follow instructions. The post-trained sibling (rivet-1b-it), which chats and reasons in `<think>` blocks, is not currently released.
|
||
|
|
|
||
|
|
## What this is for
|
||
|
|
|
||
|
|
- Fine-tuning your own model on top of a from-scratch 1B with a 16K context
|
||
|
|
- Research into small-model pretraining dynamics (full pipeline and data disclosure in the [technical report](https://huggingface.co/josh-a/rivet-1b-pt/blob/main/TECHNICAL_REPORT.md))
|
||
|
|
- A clean, fully-documented provenance chain: every pretraining token is public data or documented
|
||
|
|
|
||
|
|
## Training lineage
|
||
|
|
|
||
|
|
1. **v6 pilot**: 6.3B tokens, deliberately undertrained (2x RTX 5090)
|
||
|
|
2. **v7 main pretrain**: ~100B tokens over 7 weeks (from the 283.4B-token corpus), eval loss 1.598 (2x RTX 5090)
|
||
|
|
3. **v8 context extension**: RoPE ABF (theta 100K → 1e6) to 16K context, +3B long-doc tokens (2x H100)
|
||
|
|
4. **v9 "intelligence run"**: +20B tokens reweighted toward reasoning/math/code, eval loss 1.477. **This checkpoint.**
|
||
|
|
|
||
|
|
Total: ~129B tokens trained from the 387.5B-token pool (full composition in the [technical report](https://huggingface.co/josh-a/rivet-1b-pt/blob/main/TECHNICAL_REPORT.md)).
|
||
|
|
|
||
|
|
## Training logs
|
||
|
|
|
||
|
|
Raw TensorBoard event files for the v7 main pretrain and v9 intelligence run are in `tensorboard/` (the report's loss-curve figure is generated from these).
|
||
|
|
|
||
|
|
## Model details
|
||
|
|
|
||
|
|
| | |
|
||
|
|
|---|---|
|
||
|
|
| Params | 1,086,422,528 |
|
||
|
|
| Architecture | Qwen3ForCausalLM (from-scratch weights) |
|
||
|
|
| Hidden / layers | 2048 / 22 |
|
||
|
|
| Heads | 16 query / 8 KV (GQA), head_dim 128 |
|
||
|
|
| FFN | 5504 (SwiGLU) |
|
||
|
|
| Vocab | 32000 (custom SentencePiece) |
|
||
|
|
| Context | 16384 (RoPE θ=1e6; YaRN-extrapolable to ~64K) |
|
||
|
|
| Tokenizer | custom 32k; `<think>`=4, `</think>`=5, `<|im_start|>`=6, `<|im_end|>`=7 |
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
import torch
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
m = "josh-a/rivet-1b-pt"
|
||
|
|
tok = AutoTokenizer.from_pretrained(m)
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(m, dtype=torch.bfloat16).to("cuda").eval()
|
||
|
|
|
||
|
|
ids = tok("The capital of New Zealand is", return_tensors="pt").to("cuda")
|
||
|
|
out = model.generate(**ids, max_new_tokens=20, do_sample=False)
|
||
|
|
print(tok.decode(out[0][ids["input_ids"].shape[1]:]))
|
||
|
|
# -> " Wellington" (as of the v9 intelligence run)
|
||
|
|
```
|
||
|
|
|
||
|
|
## Evals
|
||
|
|
|
||
|
|
Per-question GSM8K results are in `evals/gsm8k_v10_results.jsonl` (1,319 rows: question, gold, prediction, raw output tail). Note: GSM8K was run on the post-trained sibling checkpoint (rivet-1b-it, not released), scoring 12.7% (168/1319). Methodology in the technical report.
|
||
|
|
|
||
|
|
## Limitations
|
||
|
|
|
||
|
|
- This is a **base model**: no instruction following, no chat template behaviour, no refusals, no alignment of any kind.
|
||
|
|
- Factually modest (1B parameters); see the technical report for the honest eval picture.
|
||
|
|
|
||
|
|
## License
|
||
|
|
|
||
|
|
Apache-2.0. From-scratch weights; no upstream model license applies.
|