145 lines
7.0 KiB
Markdown
145 lines
7.0 KiB
Markdown
# Reproducibility bundle — SZL-Khipu-1.5B-BrainNavigator
|
|
|
|
This directory is the **honest, pinned, hashed** reproduction bundle for the
|
|
Khipu navigator. It mirrors the *public* evaluation harness at a pinned commit,
|
|
lists every file with its `sha256`, and gives three graded levels of
|
|
reproduction: **verify → re-run → reforge**.
|
|
|
|
- Pinned source: `github.com/szl-holdings/szl-forge` @ `main`
|
|
`60fbe85bb4bd02ca6dbcac2db069a058d88dfe8a` (full 40-hex SHA).
|
|
- Every mirrored file's exact bytes, source repo, pinned SHA, and source path
|
|
are recorded in [`MANIFEST.json`](./MANIFEST.json). The two signed receipts and
|
|
`owner_pubkey.json` live in this model repo's **root** (not copied here) and
|
|
are listed in the manifest with their `sha256` for L1.
|
|
- Evidence labels used throughout: **REPORTED** (owner-run), **AGENT-RUN** (run
|
|
by the bundle agent, exact command shown), **RECORDED** (replayed public
|
|
bytes, not a live fetch), **DERIVED** (computed from other data), **MODELED**
|
|
(mechanism modeled, not proved).
|
|
|
|
## What is NOT public
|
|
|
|
The raw per-case model outputs from the owner's signed evaluation run are not
|
|
published — only the aggregate counts are bound in `eval_receipt.signed.json`
|
|
(`planValid` 11/11, `groundingCorrect` 4/5, `abstainCorrect` 2/6,
|
|
`hallucinatedCitationCount` 0); the signed run's per-case transcripts cannot be
|
|
independently reconstructed from this repository.
|
|
|
|
---
|
|
|
|
## L1 — Verify the signed receipts offline (no network, no GPU)
|
|
|
|
This proves **signer continuity and tamper-evidence** of the committed receipts
|
|
against the repo-declared Ed25519 key. It does **not** independently validate
|
|
model quality, data provenance, or deployment — that is a separate claim.
|
|
|
|
**Mechanism (as implemented in `szl-forge/khipu/sign_receipt.py`, re-implemented
|
|
here independently in [`verify_receipts.py`](./verify_receipts.py)):**
|
|
|
|
- The signature is Ed25519 over the receipt's `canonical` JSON string (UTF-8).
|
|
- Canonical JSON = recursively **sorted object keys**, compact separators
|
|
(`,`/`:`), **floats refused** (doctrine: strings + ints only).
|
|
- `keyId` = first **16 hex** chars of `sha256(SPKI DER of the public key)`.
|
|
- The eval→training **chain**: `eval.trainingReceiptSha256` must equal
|
|
`sha256(training receipt's canonical string)`.
|
|
|
|
```bash
|
|
# from the model repo root (so the receipts + owner_pubkey.json are present):
|
|
pip install cryptography
|
|
python3 repro/verify_receipts.py \
|
|
--pubkey owner_pubkey.json \
|
|
--training training_receipt.signed.json \
|
|
--eval eval_receipt.signed.json
|
|
```
|
|
|
|
`verify_receipts.py` re-derives the `keyId` from the SPKI (does not trust the
|
|
declared value), verifies both signatures, confirms the embedded `canonical`
|
|
string reproduces an independent canonicalization of `payload`, and recomputes
|
|
the eval→training hash chain. On success it prints a final `PASS` line and exits
|
|
`0`; any failed check prints `[XX]` and exits non-zero.
|
|
|
|
**AGENT-RUN (2026-07-16):** the command above was executed against the three
|
|
real files in this model repo. Result: `keyId` re-derived as
|
|
`89540347a69b789e` (matches the declared key), both Ed25519 signatures valid,
|
|
both canonical strings reproduced byte-identical, chain
|
|
`eval.trainingReceiptSha256 == sha256(training canonical) ==
|
|
242b52435df315aecefb42f0ae1f87bf10acc83f9af42976986b4f0f34efc081`. Final line:
|
|
**PASS** (exit 0).
|
|
|
|
The receipt-recorded dataset hashes also cross-check against the mirrored bytes
|
|
here (**DERIVED**): `eval.jsonl` `61ede148…`, `adversarial.jsonl` `812a23b3…`,
|
|
`khipu.schema.json` `b95f9927…` — all equal to the `datasets` entries inside the
|
|
signed receipts.
|
|
|
|
---
|
|
|
|
## L2 — Re-run the public harness against a local runtime (llama.cpp / GGUF)
|
|
|
|
This runs the **public** eval harness (`eval_khipu.py`) against a **quantized**
|
|
local runtime. It is a legitimate independent re-run of the harness, but it is a
|
|
**different artifact** from the owner's signed evaluation.
|
|
|
|
> **Quantization changes numerics. Your counts will NOT match the signed eval
|
|
> receipt (11/11, 4/5, 2/6). That is expected — the receipt covers the owner's
|
|
> non-quantized served artifact; a Q4_K_M GGUF is a different artifact.** Do not
|
|
> present L2 numbers as the signed eval's numbers.
|
|
|
|
The harness talks to an **Ollama** endpoint (`OLLAMA_URL`, default
|
|
`http://localhost:11434`, `/api/chat`). The GGUF quants are Ollama-ready. A
|
|
llama.cpp `llama-server` also works if you front it with an Ollama-compatible
|
|
`/api/chat` shim; the simplest honest path is Ollama below.
|
|
|
|
```bash
|
|
# 1. pull the Q4_K_M quant (Ollama-ready GGUF repo):
|
|
ollama run hf.co/SZLHOLDINGS/SZL-Khipu-1.5B-GGUF:Q4_K_M # then /bye
|
|
# equivalently with llama.cpp directly:
|
|
# huggingface-cli download SZLHOLDINGS/SZL-Khipu-1.5B-GGUF \
|
|
# <file>.Q4_K_M.gguf --local-dir ./gguf
|
|
# ./llama-server -m ./gguf/<file>.Q4_K_M.gguf --port 8080
|
|
|
|
# 2. lay the harness out as it expects (files resolve relative to the script):
|
|
# the mirrored files in repro/ are byte-identical to szl-forge/khipu/ @ the pinned SHA.
|
|
pip install jsonschema
|
|
export OLLAMA_URL=http://localhost:11434
|
|
python3 repro/eval_khipu.py # served model name defaults to "khipu"
|
|
```
|
|
|
|
Notes (stated plainly, not apologetics):
|
|
|
|
- `eval_khipu.py` verifies its held-out files against a curriculum
|
|
`manifest.json` and requires a local `training_receipt.signed.json` **before**
|
|
scoring, and fails loud if either is missing or drifted. The curriculum
|
|
`manifest.json` is part of the szl-forge working tree and is **not** mirrored
|
|
here (it references owner-side generation); to run L2 unmodified, check out
|
|
`szl-forge/khipu/` at the pinned SHA. The harness bytes themselves are mirrored
|
|
here so you can diff them against your checkout.
|
|
- The served model name the harness expects is `khipu`
|
|
(`ollama cp hf.co/SZLHOLDINGS/SZL-Khipu-1.5B-GGUF:Q4_K_M khipu` gives it that
|
|
tag).
|
|
- **Worked example (RECORDED):** `repro/agent-run-2026-07-16/` holds three
|
|
recorded L2-style traces (llama.cpp CPU, Q4_K_M, temperature=0 seed=0, exact
|
|
repro commands inside each file) — and honestly, one of them
|
|
(`adversarial-abstain-5.json`) is a **recorded abstention FAILURE** (the
|
|
quantized model emitted NAVIGATE where it should have ABSTAINed). These are
|
|
RECORDED/AGENT-RUN outputs, not the signed-receipt artifact.
|
|
|
|
---
|
|
|
|
## L3 — Full owner-metal reproduction (train + sign) — owner-side
|
|
|
|
The complete forge (QLoRA training, curriculum generation, receipt signing on
|
|
owner hardware) is documented in **`RUNBOOK-KHIPU.md`** in `szl-forge` at the
|
|
pinned SHA:
|
|
|
|
`https://github.com/szl-holdings/szl-forge/blob/60fbe85bb4bd02ca6dbcac2db069a058d88dfe8a/khipu/RUNBOOK-KHIPU.md`
|
|
|
|
This path is **owner-side**: it requires the owner's training rig and the
|
|
owner's private signing key (which never leaves owner hardware). Only the owner
|
|
can regenerate the signed receipts; third parties reproduce at L1 (verify) and
|
|
L2 (re-run the public harness on a local runtime). L3 is a pointer, not a claim
|
|
that anyone else can re-sign.
|
|
|
|
---
|
|
|
|
_Part of the SZL-Forge family. `a-11-oy.com`. Receipt integrity is verifiable
|
|
from the committed files (L1); runtime/deployment status is a separate claim._
|