license, language, pipeline_tag, tags, library_name

license	language	pipeline_tag	tags	library_name
apache-2.0	en	text-generation	text-generation pretrained base-model from-scratch qwen3	transformers

Rivet-1B-PT (base model)

The pretrained base of Rivet-1B: a 1.09B parameter language model trained entirely from scratch (custom tokenizer, custom corpus, no inherited weights). This repo holds the raw pretrained checkpoint (v9 lineage), before any supervised fine-tuning.

Uses the Qwen3 architecture for serving compatibility. Shares no weights with Qwen: the tokenizer (custom 32k SentencePiece) and every parameter were trained from zero.

Note: this is the raw base model. It completes text; it does not follow instructions. The post-trained sibling (rivet-1b-it), which chats and reasons in <think> blocks, is not currently released.

What this is for

• Fine-tuning your own model on top of a from-scratch 1B with a 16K context
• Research into small-model pretraining dynamics (full pipeline and data disclosure in the technical report)
• A clean, fully-documented provenance chain: every pretraining token is public data or documented

Training lineage

1. v6 pilot: 6.3B tokens, deliberately undertrained (2x RTX 5090)
2. v7 main pretrain: ~100B tokens over 7 weeks (from the 283.4B-token corpus), eval loss 1.598 (2x RTX 5090)
3. v8 context extension: RoPE ABF (theta 100K → 1e6) to 16K context, +3B long-doc tokens (2x H100)
4. v9 "intelligence run": +20B tokens reweighted toward reasoning/math/code, eval loss 1.477. This checkpoint.

Total: ~129B tokens trained from the 387.5B-token pool (full composition in the technical report).

Training logs

Raw TensorBoard event files for the v7 main pretrain and v9 intelligence run are in tensorboard/ (the report's loss-curve figure is generated from these).

Model details

Params	1,086,422,528
Architecture	Qwen3ForCausalLM (from-scratch weights)
Hidden / layers	2048 / 22
Heads	16 query / 8 KV (GQA), head_dim 128
FFN	5504 (SwiGLU)
Vocab	32000 (custom SentencePiece)
Context	16384 (RoPE θ=1e6; YaRN-extrapolable to ~64K)
Tokenizer	custom 32k; <think>=4, </think>=5, `<

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

m = "josh-a/rivet-1b-pt"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m, dtype=torch.bfloat16).to("cuda").eval()

ids = tok("The capital of New Zealand is", return_tensors="pt").to("cuda")
out = model.generate(**ids, max_new_tokens=20, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:]))
# -> " Wellington" (as of the v9 intelligence run)

Evals

Per-question GSM8K results are in evals/gsm8k_v10_results.jsonl (1,319 rows: question, gold, prediction, raw output tail). Note: GSM8K was run on the post-trained sibling checkpoint (rivet-1b-it, not released), scoring 12.7% (168/1319). Methodology in the technical report.

Limitations

• This is a base model: no instruction following, no chat template behaviour, no refusals, no alignment of any kind.
• Factually modest (1B parameters); see the technical report for the honest eval picture.

License

Apache-2.0. From-scratch weights; no upstream model license applies.