初始化项目，由ModelHub XC社区提供模型

Model: josh-a/rivet-1b-pt
Source: Original Platform

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+paper.pdf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,82 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+tags:
+- text-generation
+- pretrained
+- base-model
+- from-scratch
+- qwen3
+library_name: transformers
+---
+
+# Rivet-1B-PT (base model)
+
+The **pretrained base** of Rivet-1B: a 1.09B parameter language model trained **entirely from scratch** (custom tokenizer, custom corpus, no inherited weights). This repo holds the raw pretrained checkpoint (v9 lineage), before any supervised fine-tuning.
+
+> Uses the Qwen3 *architecture* for serving compatibility. Shares **no weights** with Qwen: the tokenizer (custom 32k SentencePiece) and every parameter were trained from zero.
+
+**Note:** this is the raw base model. It completes text; it does not follow instructions. The post-trained sibling (rivet-1b-it), which chats and reasons in `<think>` blocks, is not currently released.
+
+## What this is for
+
+- Fine-tuning your own model on top of a from-scratch 1B with a 16K context
+- Research into small-model pretraining dynamics (full pipeline and data disclosure in the [technical report](https://huggingface.co/josh-a/rivet-1b-pt/blob/main/TECHNICAL_REPORT.md))
+- A clean, fully-documented provenance chain: every pretraining token is public data or documented
+
+## Training lineage
+
+1. **v6 pilot**: 6.3B tokens, deliberately undertrained (2x RTX 5090)
+2. **v7 main pretrain**: ~100B tokens over 7 weeks (from the 283.4B-token corpus), eval loss 1.598 (2x RTX 5090)
+3. **v8 context extension**: RoPE ABF (theta 100K → 1e6) to 16K context, +3B long-doc tokens (2x H100)
+4. **v9 "intelligence run"**: +20B tokens reweighted toward reasoning/math/code, eval loss 1.477. **This checkpoint.**
+
+Total: ~129B tokens trained from the 387.5B-token pool (full composition in the [technical report](https://huggingface.co/josh-a/rivet-1b-pt/blob/main/TECHNICAL_REPORT.md)).
+
+## Training logs
+
+Raw TensorBoard event files for the v7 main pretrain and v9 intelligence run are in `tensorboard/` (the report's loss-curve figure is generated from these).
+
+## Model details
+
+| | |
+|---|---|
+| Params | 1,086,422,528 |
+| Architecture | Qwen3ForCausalLM (from-scratch weights) |
+| Hidden / layers | 2048 / 22 |
+| Heads | 16 query / 8 KV (GQA), head_dim 128 |
+| FFN | 5504 (SwiGLU) |
+| Vocab | 32000 (custom SentencePiece) |
+| Context | 16384 (RoPE θ=1e6; YaRN-extrapolable to ~64K) |
+| Tokenizer | custom 32k; `<think>`=4, `</think>`=5, `<|im_start|>`=6, `<|im_end|>`=7 |
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+m = "josh-a/rivet-1b-pt"
+tok = AutoTokenizer.from_pretrained(m)
+model = AutoModelForCausalLM.from_pretrained(m, dtype=torch.bfloat16).to("cuda").eval()
+
+ids = tok("The capital of New Zealand is", return_tensors="pt").to("cuda")
+out = model.generate(**ids, max_new_tokens=20, do_sample=False)
+print(tok.decode(out[0][ids["input_ids"].shape[1]:]))
+# -> " Wellington" (as of the v9 intelligence run)
+```
+
+## Evals
+
+Per-question GSM8K results are in `evals/gsm8k_v10_results.jsonl` (1,319 rows: question, gold, prediction, raw output tail). Note: GSM8K was run on the post-trained sibling checkpoint (rivet-1b-it, not released), scoring 12.7% (168/1319). Methodology in the technical report.
+
+## Limitations
+
+- This is a **base model**: no instruction following, no chat template behaviour, no refusals, no alignment of any kind.
+- Factually modest (1B parameters); see the technical report for the honest eval picture.
+
+## License
+
+Apache-2.0. From-scratch weights; no upstream model license applies.

TECHNICAL_REPORT.md

@@ -0,0 +1,256 @@
+# Rivet-1B: A From-Scratch 1.09B Reasoning Model
+
+**Josh Angel** · [LinkedIn](https://www.linkedin.com/in/j0sha/) · [Hugging Face](https://huggingface.co/josh-a)
+**August 2026**
+
+Released: [josh-a/rivet-1b-pt](https://huggingface.co/josh-a/rivet-1b-pt) (pretrained base + this report). Not released: the post-trained model (rivet-1b-it) and training datasets (rivet-sft, rivet-cpt).
+
+---
+
+## Abstract
+
+Rivet-1B is a 1.09B parameter reasoning language model trained entirely from scratch: custom tokenizer, custom pretraining corpus, custom post-training. No weights, embeddings, or tokenizer state were inherited from any existing model. It uses the Qwen3 architecture purely for serving compatibility and shares nothing with Qwen beyond the config schema.
+
+The model was built as the reasoning core of a personal agent: it thinks in dedicated `<think>` blocks, drafts, rewrites, summarises, triages, and writes short code in a specific voice. Out-of-pocket spend was limited to a short burst of rented H100 compute for context extension and teacher-model API calls for distillation. The bulk of training, approximately 130B tokens of pretraining and continued pretraining, ran on owned 2x RTX 5090 hardware at zero marginal cost.
+
+Against a 30-task held-out release bar, the released checkpoint (v10) scores 22/30: strong on directed task execution, identity, and short reasoning; weak on factual recall and multi-step arithmetic, which I show is a parameter-count ceiling rather than a data-mix problem. Needle-in-a-haystack retrieval is 100% at the native 16K context and at 64K via YaRN extrapolation.
+
+This report documents the full pipeline, the data recipes, the eval methodology, and the engineering failures that shaped the result, including a silently 58%-empty SFT corpus, a KV-cache bug that invalidated an entire round of eval conclusions, and a multi-node experiment that halved throughput.
+
+## 1. Why build a 1B from scratch
+
+The practical question: how much useful capability can one person put into a model they fully own, on hardware they already have, for pocket change?
+
+Fine-tuning an existing open model would have been cheaper and immediately stronger. It also would have meant inheriting someone else's tokenizer, data decisions, license posture, and unknown pretraining mixture. The goals here were different:
+
+1. **Full provenance.** Every token the model saw is known and documented. For a personal agent trained partly on private communications, that matters.
+2. **A reasoning-native format.** `<think>` and `</think>` are dedicated single tokens in the vocabulary (IDs 4 and 5), not strings learned post-hoc. Reasoning is part of the model's native grammar.
+3. **A specific voice and persona**, baked into weights rather than simulated with ever-longer system prompts.
+4. **The learning.** Running pretraining end-to-end surfaces failure modes you never see when fine-tuning: data-mix design, warmup dynamics, loss-plateau interpretation, context extension, and the long tail of infrastructure bugs documented in Section 8.
+
+The model is named Rivet, after the agent it powers.
+
+## 2. Architecture
+
+Rivet-1B is a decoder-only transformer in the Qwen3 family: RMSNorm pre-norm, QK-Norm, grouped-query attention, SwiGLU, RoPE, tied embeddings.
+
+| Property | Value |
+|---|---|
+| Parameters | 1,086,422,528 |
+| Hidden size | 2048 |
+| Layers | 22 |
+| Attention heads | 16 query / 8 KV (GQA 2:1) |
+| Head dimension | 128 |
+| FFN intermediate | 5504 (SwiGLU) |
+| Vocabulary | 32,000 (custom SentencePiece BPE) |
+| Context | 16,384 trained (RoPE theta 1e6), YaRN-extrapolable to ~64K |
+| Embeddings | Tied (lm_head = embed_tokens, saves 65.5M params) |
+
+Parameter split: 65.5M embedding (6%), 1,021M across 22 transformer layers (94%).
+
+Design notes:
+
+- **QK-Norm** (RMSNorm on queries and keys per head, before RoPE) for training stability, following OLMo 2 and Qwen 3. Cost: 5,632 parameters total.
+- **Tied embeddings** as regularizer and parameter saver; standard at this scale (SmolLM2, Gemma).
+- **GQA 2:1** halves KV-cache memory at inference with negligible quality cost.
+- **Qwen3-family config** so the trained weights export mechanically to `Qwen3ForCausalLM` and serve on vLLM, llama.cpp, Ollama, and transformers with zero custom code. The export is a ~250-line key-remap; the architecture match is exact (RMSNorm pre-norm + QK-Norm + GQA + SwiGLU + RoPE + tied embeddings).
+
+## 3. Tokenizer
+
+Custom SentencePiece BPE, 32,000 vocab, trained on 10M sentences sampled from the pretraining corpus, byte fallback enabled.
+
+Special tokens are first-class vocabulary entries:
+
+| ID | Token | Role |
+|---|---|---|
+| 0 | `<unk>` | unknown |
+| 1 | `<s>` | BOS |
+| 2 | `</s>` | EOS |
+| 3 | `<pad>` | padding |
+| 4 | `<think>` | reasoning block start |
+| 5 | `</think>` | reasoning block end |
+| 6 | `<\|im_start\|>` | ChatML turn start |
+| 7 | `<\|im_end\|>` | ChatML turn end (end-of-turn; generation stops here) |
+
+Two consequences worth noting:
+
+- Because `<think>` is a single dedicated token, the model cannot "almost" start a reasoning block. Reasoning delimiters are atomic, which made SFT format compliance effectively 100%.
+- Serving requires `eos_token_id=[7, 2]` (end-of-turn, then EOS) and care with `skip_special_tokens`, which strips the think tags by default. I serve bare ChatML with no reasoning parser and split on `</think>` client-side.
+
+## 4. Pretraining
+
+### 4.1 Corpus
+
+The assembled pool is **387.5B unique tokens** (722 GB tokenized, packed, uint16), ~99% public data plus a small slice of proprietary English data I built (<0.2%, not redistributed).
+
+**v7 pretraining corpus, 283.4B tokens:**
+
+| Source | Tokens | Sampling weight | Role |
+|---|---|---|---|
+| DCLM-Edu | 170B | 40% | filtered educational web (SmolLM2 recipe) |
+| FineMath 3+ | 41B | 15% | math reasoning |
+| Cosmopedia v2 | 30B | 20% | synthetic textbook |
+| codeparrot-clean | 27B | 8% | Python code |
+| FineWeb-Edu 10BT | 10B | 7% | educational web |
+| Wikipedia EN | 5B | 7% | factual grounding |
+| Proprietary English v2 | 0.28B | 3% | domain data I built, oversampled |
+
+**v8 continued-pretraining increment, 104.1B tokens:** multi-language code (69B, StarCoderData), books (15B, Project Gutenberg), reasoning (10B, OpenThoughts / OpenR1), conversations (9B, StackExchange + GitHub issues), Proprietary English v3 (0.38B).
+
+Mix design followed published small-model recipes (SmolLM2, Phi, Hermes), validated against ablations rather than guessed. The full composition is documented below; raw bins are not re-uploaded since the sources are public.
+
+### 4.2 Runs
+
+Training used a custom `RivetForCausalLM` wrapper under the HuggingFace Trainer with a WSD schedule (warmup 2% / stable 78% / decay 20%), AdamW, bf16, gradient checkpointing, sequence packing, DDP.
+
+| Run | Tokens | Hardware | Wall time | Final eval loss |
+|---|---|---|---|---|
+| v6 (initial) | 6.3B | 2x RTX 5090 | ~52h | 2.86 |
+| v7 (main pretrain) | ~100B (of the 283.4B corpus) | 2x RTX 5090 | ~7 weeks | 1.598 |
+| v8 CPT (context ext.) | +3B long-doc | 2x H100 (rented) | ~days | ~0.97 |
+| v9 ("intelligence run") | +20B reweighted | 2x RTX 5090 | 10.9 days, 38,146 steps, 0 failures | 1.477 |
+
+Narrative arc:
+
+- **v6** was deliberately treated as a pilot: 6.3B tokens (~5.8x params) is far below Chinchilla-optimal, and the model was predictably undertrained. Its value was as a warm-start init: v7's first eval at step 5K was 1.898, where a from-zero run sits at 6-7.
+- **v7** was the main event: 1,220,703 steps at ~24K tokens/sec aggregate (3.4 s/step, both 5090s pinned ~97%), seven weeks of continuous uptime. Eval loss descended through a series of plateaus (1.92 at 148K steps, 1.85 at 295K) with the LR decay phase delivering the final drop to 1.598.
+- **v8 CPT** extended context from 2048 to 16K via RoPE ABF (theta 100K to 1e6) on a long-document mix. This was the one rented-compute run: 2x H100 on Brev. (Its ~0.97 eval loss is measured on the long-doc eval set and is not comparable to the v7/v9 figures.)
+- **v9** warm-started v8 and sampled +20B tokens from the pool reweighted toward reasoning/math/code. This measurably raised the knowledge floor: base-completion probes went from garbage ("capital of New Zealand" completed to "£100,000,000") to correct ("Wellington"), and stayed correct through SFT.
+
+Throughput anchor: **24K tokens/sec aggregate (~2B tokens/day)** for the 1B on 2x 5090.
+
+### 4.3 What pretraining at this scale actually teaches
+
+A consistent pattern across checkpoint A/B probes (seeds pinned, greedy and sampled): the model learns to *say something English* long before it learns to *say something correct*. Broken-output rates dropped steadily from step 148K to 324K while factual accuracy stayed flat, then jumped only after tens of billions more tokens. Eval loss was the only monotonic signal; greedy spot-checks of individual facts were anecdote-grade noise on the ±5K-step scale.
+
+## 5. Context extension
+
+Native context is 16,384 tokens, trained via RoPE ABF (theta raised 100K to 1e6) during the v8 CPT run on long documents.
+
+Validation: needle-in-a-haystack retrieval is **100% at 16K native and at 64K via YaRN** (factor 4 extrapolation at inference, no further training).
+
+One operational gotcha worth recording: checkpoints repeatedly saved a stale `max_position_embeddings=2048` from an early config bug. The fingerprint of a correct checkpoint is `rope_theta=1e6` with `max_position_embeddings=16384`; I now verify both on every export.
+
+## 6. Post-training
+
+### 6.1 Format and masking
+
+SFT data is ShareGPT-format conversations. Every assistant turn is a `<think>` block (real planning, capped short) followed by the finished artifact. Loss is computed on assistant tokens only (prompt tokens masked to -100), multi-turn supported with each assistant turn unmasked individually. Chat rendering is bare ChatML through the HF tokenizer's `apply_chat_template`, so SFT tokenization is bit-identical to what vLLM and llama.cpp produce at inference.
+
+### 6.2 The v10 corpus
+
+The released model's SFT corpus is 42,066 weight-expanded rows from 33,477 unique:
+
+| Component | Unique | Weight | Purpose |
+|---|---|---|---|
+| Distilled task-completion | 3,944 | x3 | the core signal: draft / rewrite / summarise / triage / code / decisions / math |
+| Reasoning + math with real `<think>` | 35,000 | x1 | general reasoning |
+| Identity seed | 39 | x20 | persona |
+
+A Rivet system prompt is baked into ~70% of rows so the persona lives in the weights; 30% are system-free so the model still behaves bare. Task signal is ~21% of the final mix.
+
+The distilled task-completion set was purpose-built: ~4,650 prompts across seven task clusters, answered by Claude Sonnet under an instruction enforcing direct voice, no customer-service filler, finished artifacts only, and short substantive think traces. Earlier distillation rounds produced the "apex" reasoning tier: 3,155 examples across 27 categories with real extended-thinking traces from Claude Opus 4.6 via the Batch API.
+
+### 6.3 What the data iterations taught
+
+The path from SFT v8 to v10 was driven by eval failures, and each fix was a *data composition* fix, not a hyperparameter fix:
+
+- **v8** scored eval_loss 0.251 but felt flat. Root cause found later: the corpus it trained on was **58% empty rows** (124,955 of 214,351), written as structural placeholders by a reformatting bug. They passed the loader's `len(convs) < 2` check because empty rows still had two structural turns. Validation must check non-empty *content*, not structure.
+- **v9-SFT** scored 10/30 on the release bar. Failure clusters: task-execution ramble (meta-commentary instead of artifacts), verbose think traces eating the answer budget, word-problem math collapse, hallucinated context, and a hard safety fail (complied with a phishing-email request).
+- **v10** cut generic instruction filler (alpaca, helpdesk-register data) that taught deflection, added the distilled task-completion set, capped think length in the data, and baked the persona into 70% of rows. Result: 22/30, with task-execution, identity, and short reasoning all passing.
+- **v11** attempted to recover factual recall by rebalancing the mix toward knowledge data. It scored 19/30 and facts got *worse*. Conclusion: factual weakness at 1B is a parameter ceiling, not an SFT-mix problem. v11 was discarded; v10 stands.
+
+The refusal question was decided explicitly: the v10 corpus is **unfiltered**. A 40-example refusal set was built and then deliberately excluded. Rivet is a personal model that attempts any request; the deployer owns safety. This is called out on the model card in the first screen of text.
+
+## 7. Evaluation
+
+### 7.1 Release bar
+
+The primary gate is a 30-task held-out battery (`release_bar.jsonl`) spanning identity, drafting, rewriting, summarising, triage, code, reasoning, facts, multi-turn recall, format, voice, and refusal behavior. Scored greedy, temperature 0, with the production system prompt.
+
+| Checkpoint | Score |
+|---|---|
+| v9-SFT (baseline) | 10/30 |
+| **v10 (released)** | **22/30** |
+| v11 (rebalance experiment) | 19/30 (discarded) |
+
+v10 passes: identity (3/3), all drafting/rewriting/summarising tasks, triage, short code (fib, bash), single-step arithmetic, decisions, format, multi-turn recall, voice.
+
+v10 fails: factual recall (NZ capital answered "Canterbury"), multi-step word problems (arrival-time, work-rate), one tool-explanation task, and two cutoff-adjacent tasks.
+
+### 7.2 Base-model knowledge floor
+
+Continued pretraining's effect was verified directly on base completions (no chat template): "The capital of New Zealand is" went from numeric garbage (v8) to "Wellington" (v9). This matters because it isolates pretraining gains from SFT presentation gains.
+
+### 7.3 GSM8K
+
+On the full GSM8K test set (1,319 problems, greedy decoding, chat format with the production system prompt, answers extracted after `</think>`, with any unparseable generation re-run individually to rule out batching artifacts), the released checkpoint achieves **12.7% (168/1319)**. Per-question results ship with the model (`evals/gsm8k_v10_results.jsonl` in the repo). This sits in the expected band for a 1B model trained on 130B tokens, and is consistent with the release bar's weakest cluster (multi-step word problems). GSM8K measures precisely the capability this model is not built for; the release bar in 7.1 measures the one it is.
+
+Set against published 1B-class baselines, the number reads differently:
+
+| Model | Trained tokens | Greedy | pass@4 |
+|---|---|---|---|
+| Pythia-1B | 300B | 0.0% | 3.7% |
+| Pythia-1.4B | 300B | 1.3% | 4.0% |
+| TinyLlama-1.1B | 3T | 0.7% | 6.3% |
+| OLMo-2-1B | 4T | 0.7% | 11.3% |
+| SmolLM2-1.7B | 11T | 4.0% | 14.7% |
+| **Rivet-1B** | **130B** | **13.7%** | **25.7%** |
+| Qwen2.5-1.5B | 18T | 51.7% | 76.7% |
+
+Every model run by me under one uniform harness on a fixed 300-problem test subset (seed 42), 0-shot, no few-shot examples. Greedy: single deterministic generation. pass@4: correct within four temp-0.7 samples. Per-question results for every model in `evals/`. Published baseline figures (Pythia ~2%, TinyLlama 1.4% 5-shot) are consistent with these runs. Qwen2.5-1.5B (math-saturated 18T-token corpus) sits well above the field and is shown for completeness; one third-party report of Qwen2.5-1.5B base at 2.1% zero-shot is a harness artifact and did not reproduce. Rivet-1B additionally scores 12.7% greedy on the full 1,319-problem test set.
+
+**Contamination disclosure.** Rivet-1B's SFT corpus contains 2,635 rows drawn from the GSM8K *train* split (verified by exact and prefix match over all 41,696 SFT rows), about 6% of the mix. Training on the train split is standard practice; the *test* split was verified absent (zero matches), so the scores above measure generalization from train to test in the conventional way. I state this so nobody has to catch it.
+
+**Variance study** (same 300-problem subset; per-condition results in `evals/gsm8k_variance_results.jsonl`): a single greedy number understates the model. Four samples at temp 0.7 give avg@1 11.2% but pass@4 **25.7%**; removing the system prompt changes nothing (13.3% greedy); removing the `####` format instruction nearly doubles the score to **21.0%**, because unconstrained generations run their think trace to completion and end with a natural-language answer, while the format instruction truncates or derails a third of generations. Reading: the point estimate is stable, the ceiling is higher than any single greedy run shows, and the format instruction is a liability on math, worth fixing in the next SFT round.
+
+### 7.4 Long context
+
+NIAH: 100% retrieval at 16K native, 100% at 64K under YaRN (factor 4).
+
+### 7.4 Honest summary
+
+Rivet-1B is sharp at *doing*: given a task, it produces a clean artifact in the right voice with a visible reasoning trace. It is modest at *knowing*: at 1.09B parameters it is not a knowledge base and will confidently produce plausible-wrong facts. The eval program's job was to make that tradeoff explicit and measured rather than vibes.
+
+## 8. Engineering failures that shaped the result
+
+The most useful part of this report for anyone replicating the path:
+
+1. **The 58%-empty SFT corpus.** A reformat step wrote `{"conversations": []}` placeholders for rows it couldn't convert. The loader's structural check passed them. An entire SFT round trained on it. Fix: validate content, not structure; count empty rows in the build script and fail loudly.
+2. **The KV-cache bug that invalidated a month of conclusions.** The custom model's `forward()` returned `CausalLMOutput` instead of `CausalLMOutputWithPast`, breaking HF `generate()`'s cache contract. Output looked like garbage, and "SFT is broken" was the working theory until manual greedy decoding produced fluent reasoning. The model was fine; the decode path was not. Rather than retrofit KV cache through every custom attention layer, I exported to standard Qwen3 format and deleted the problem.
+3. **The tied-weights safetensors crash.** transformers v5 requires `_tied_weights_keys` declared (as a dict; v4 wanted a list) plus `post_init()` and config mirroring of `tie_word_embeddings`. First checkpoint save crashed at step 2000 on rented H100s. A lesson paid for in rented compute.
+4. **Multi-node is not free.** 4x 5090 across two machines over 10 GbE ran at *half* the throughput of single-node 2x 5090: bf16 gradient all-reduce of ~2 GB/step over TCP dominated. At 1B params, compute does not outweigh comms until ~100 GbE interconnects. Killed the experiment, stayed single-node.
+5. **Infinite eval.** An `IterableDataset` with `while True: yield` was reused for eval; the Trainer iterated faithfully forever. First eval ran 4.5 hours before being killed. Fix: materialize a fixed eval subset.
+6. **Loss logging inflation.** transformers v5 logs train loss multiplied by `grad_accum x world_size` (20x in my config). A run that looked broken (loss "187") was healthy (~9.3). Fixed with a `Trainer.log()` override.
+7. **Stale config on save.** `max_position_embeddings=2048` persisted into 16K-trained checkpoints. Verify config fingerprints on every checkpoint, not just weights.
+
+None of these are exotic. All of them cost real time, and none appear in fine-tuning-a-Llama tutorials because that path never exercises them.
+
+## 9. Cost
+
+I deliberately don't quote a headline dollar figure. Any such number is arbitrary: it depends entirely on how you account for owned hardware, power, and time, and every convention produces a different answer. What I can state precisely is the shape of the spend. The dominant cost was patience, not money: roughly 130B tokens of pretraining and continued pretraining, plus every SFT round, ran on owned GPUs (2x RTX 5090, already paid for) at zero marginal cost over several months. Cash spend was limited to the two things owned hardware couldn't provide: a few days of rented H100 capacity for the 16K context-extension run, and teacher-model API calls for the distillation sets. Both were small relative to the compute they unlocked.
+
+## 10. Limitations
+
+- **Facts are unreliable.** This is the 1B parameter ceiling, demonstrated experimentally (Section 6.3, v11). Do not use Rivet-1B as a knowledge source.
+- **Multi-step arithmetic and word problems** are unreliable.
+- **Unfiltered.** No refusal or safety training, by design. The deployer owns that layer.
+- **Personal-data provenance.** SFT included the author's own writing (voice/persona data). That persona lives only in the post-trained model, which is not released. The released base model was never exposed to it.
+- **English-only**, by corpus design.
+
+## 11. What's next
+
+The 1B is released and frozen. Current work is a **~3B growth model**: block expansion (LLaMA-Pro-style, function-preserving) carrying the trained 1B forward, then continued pretraining on the 387.5B-token pool already on disk, then re-SFT. The motivation is exactly the v11 lesson: facts and multi-step reasoning are parameter-limited, and block expansion inherits ~130B tokens of training instead of starting over.
+
+That model is unreleased and not covered by this report.
+
+## 12. Reproduce
+
+- Pretrained base weights + model card + raw TensorBoard logs (`tensorboard/`) + GSM8K results (`evals/`): [josh-a/rivet-1b-pt](https://huggingface.co/josh-a/rivet-1b-pt)
+- The post-trained model (rivet-1b-it) and training datasets (rivet-sft, rivet-cpt) are not released; their composition is documented in this report.
+
+Serving quickstart is on the model card. The two non-obvious requirements: `eos_token_id=[7, 2]`, and handle `<think>` as special tokens (no reasoning parser; split on `</think>` yourself).
+
+---
+
+*Trained in Auckland, New Zealand. Compute: 2x RTX 5090 (owned) + 2x H100 (rented). Teachers: Claude Opus 4.6 and Sonnet (distillation). Everything else: from scratch.*

chat_template.jinja

@@ -0,0 +1,4 @@
+{%- for message in messages %}{{ '<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>
+' }}{%- endfor %}{%- if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{%- endif %}

config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": [
+    7,
+    2
+  ],
+  "pad_token_id": 3,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 5504,
+  "max_position_embeddings": 16384,
+  "max_window_layers": 22,
+  "model_type": "qwen3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 22,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "5.8.0",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.8.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9971e937ef6753c01caec744b1ab2c9e53897ba06850531638b091ce4ffa10e3
+size 2172872928

paper.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bdc3aedb5be4e75ced1c8ba69109b8871fc46690534d702cb630d22b80d2f481
+size 116728

tensorboard/v7_pretrain/May15_05-51-40_gpu-rack-02/events.out.tfevents.1778781100.gpu-rack-02.1995184.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17ae3dc362786d046f1f25c2b3002b1dfd4059ead26aa417d72497bac5b2a842
+size 12752

tensorboard/v7_pretrain/May15_07-53-26_gpu-rack-02/events.out.tfevents.1778788406.gpu-rack-02.2173418.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9f01ea35c308f2b0934f18cc648c4b31031cf65862e0b3ab924bbcffad94773
+size 25412

tensorboard/v7_pretrain/May15_16-12-06_gpu-rack-02/events.out.tfevents.1778818326.gpu-rack-02.2931036.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23ced38e2d908a6136c0463c235a872d840a0124651a79c0e755f37abd73f2af
+size 5319979

tensorboard/v9_intelligence_run/Jul15_14-43-46_gpu-rack-02/events.out.tfevents.1784083426.gpu-rack-02.2390077.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd319f28ee87417c791122086bddf3ad9ca13ed07d78ee368bed31a9f75423ab
+size 4293

tensorboard/v9_intelligence_run/Jul15_14-54-33_gpu-rack-02/events.out.tfevents.1784084073.gpu-rack-02.2406257.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:323cf74a78b1e59d9a9f051d869a69a884a7828eb5b6d7c9505354f3fcea73a5
+size 4293

tensorboard/v9_intelligence_run/Jul15_15-00-43_gpu-rack-02/events.out.tfevents.1784084443.gpu-rack-02.2415939.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a8507c1694c16a9f711bff302a147a3d57cf671fb24095075c93dbf1b6d9e31
+size 4293

tensorboard/v9_intelligence_run/Jul15_15-09-56_gpu-rack-02/events.out.tfevents.1784084996.gpu-rack-02.2429638.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54414c7454d3416df7edd2e64387aa679b3d007e532d5886b47076d7a964499e
+size 414098

tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "add_prefix_space": null,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": [
+    "<think>",
+    "</think>",
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "from_slow": true,
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 2048,
+  "pad_token": "<pad>",
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}