Bonsai-8B-Ternary-RDNA4/README.md

license, base_model, tags, library_name

license	base_model	tags	library_name
apache-2.0	prism-ml/Ternary-Bonsai-8B	gguf rdna4 gfx1201 ternary speculative-decoding the-rock8 llama.cpp	gguf

Bonsai-8B Ternary — RDNA4 (The Rock8) 🦆

RDNA4 (gfx1201) GGUF builds of prism-ml/Ternary-Bonsai-8B — a dense Qwen3-8B trained natively ternary (QAT, 1.58-bit). Part of the The Rock8 — RDNA4 fp8 collection.

This repo is the async speculative-decoding showcase for The Rock8 — the config that hits +66% decode on a dual-R9700 box, byte-identical output.

Files

File	Size	Role
Ternary-Bonsai-8B-F16.gguf	16.4 GB	Verify target — ternary weights in F16 storage, runs on RDNA4 WMMA
Ternary-Bonsai-8B-Q2_0.gguf	2.18 GB	The ternary self-draft — the cheap drafter that makes async win
Ternary-Bonsai-8B-F8E4M3.gguf	8.6 GB	Native RDNA4 fp8 (E4M3) build

The async spec-decode win (measured on gfx1201, dual R9700)

Using the ternary Q2 draft to speculate for the F16 target, with The Rock8's async pipeline (LLAMA_SPEC_ASYNC=2) — draft-gen on GPU1 ‖ verify on GPU0:

Config	Decode t/s	Accept	Output
Sequential, 1-GPU	63.30 ± 0.53	100%	baseline
Async pipeline, 2-GPU	105.08 ± 0.70	100%	byte-identical ✅

+66% decode, zero quality loss (re-validated 2026-07-13 on the TheRock ROCm 7.13 build; peaks ~111 t/s / +75% on an unloaded box).

Why it works — "intel per byte"

A natively-ternary model is its own near-lossless, cheap draft. The Q2 draft runs on the VALU/mmvq path while the F16 target verifies on the WMMA tensor cores — different execution units, so draft-gen and verify genuinely overlap instead of fighting for the same silicon. That disjoint-compute overlap is the whole trick, and it's why async pays off here but not on a same-precision self-draft (draft costs as much as verify → nothing to hide).

Note: the async pipeline is a 2-GPU lever and needs a dense target (plain-attention KV supports the pipeline's partial-rollback). Hybrid SSM/GatedDeltaNet targets need a core-level rollback fix — see the collection notes.

Not Bonsai-specific — async wins on off-the-shelf models too

The pipeline is a general lever: it wins whenever the verify is heavy and the draft is cheap. Same trick, vanilla Qwen3-8B (byte-identical output in every case):

Target	Draft	Async Δ
Qwen3-8B fp8 (light)	Q4_K_M	−9% (too light — loses)
Qwen3-8B BF16 (heavy)	Q4_K_M	+14.5%
Qwen3-8B BF16 (heavy)	Q2_K	+21.9%
Bonsai F16 (heavy)	ternary Q2 (VALU)	+66%

Swapping the identical Q4 draft from an fp8 target to a BF16 target flips −9% → +14.5% — the target's verify-weight is the deciding lever, not the draft. Bonsai tops the table because its ternary draft runs entirely off the WMMA units (max overlap); a heavier target × a cheaper draft ⇒ a bigger win.

Usage (The Rock8 fork)

# async 2-GPU: ternary Q2 draft ‖ F16 verify
LLAMA_SPEC_ASYNC=2 ./llama-speculative-simple \
  -m   Ternary-Bonsai-8B-F16.gguf   -dev  ROCm0 \
  -md  Ternary-Bonsai-8B-Q2_0.gguf  -devd ROCm1 \
  --spec-type draft-simple --spec-draft-n-max 4 \
  -c 2048 --temp 0 -n 130 \
  -p "What do you call a dried grape? Answer in one word."

Build: github.com/The-Monk/The-Rock8 (RDNA4 native-fp8 llama.cpp fork + Podman appliance).

Attribution & license

Base model prism-ml/Ternary-Bonsai-8B by PrismML, Apache-2.0. These are GGUF conversions/quantizations for RDNA4; all credit for the model and its ternary QAT training to PrismML. Distributed under the same Apache-2.0 license.

🦆 Got any weights?