60 lines
3.0 KiB
Markdown
60 lines
3.0 KiB
Markdown
# ConeML 810M — evaluation methodology and variant lineage (public record)
|
||
|
||
Two released variants share one from-scratch 810M base checkpoint and
|
||
differ only in supervised fine-tuning; both were evaluated with the same
|
||
bf16 instruments.
|
||
|
||
- `coneml-810m-alpha` — function-writing / mixed-arithmetic profile
|
||
- `coneml-810m-alpha-arithmetic` — carry/borrow arithmetic variant
|
||
|
||
## Methodology
|
||
|
||
- **ConeML internal certification**: large-n (300–500 per task family)
|
||
batteries of unseen instances in the trained prompt formats, greedy
|
||
decoding, repetition
|
||
penalty 1.15. Code tasks are scored by EXECUTING the generated function
|
||
against held-out tests; SQL by executing against a live sqlite database;
|
||
scripts by captured stdout. No pattern-match scoring for anything
|
||
execution can decide.
|
||
- **Behavior probes**: held-out designated-refusal prompts plus in-scope
|
||
contrast pairs in matching wording; one separately rerun, scripted
|
||
eight-turn conversation; and an everyday-reasoning screen with manual
|
||
adjudication. Aggregate results and deterministic representative
|
||
ConeML-owned rows are public; complete rows are retained privately.
|
||
- **Public benchmarks**: GSM8K (first 200, zero-shot, strict final-number
|
||
match) and HumanEval (pass@1, tests executed), identical harness across
|
||
variants and the pretrained base. Aggregate results are public;
|
||
benchmark rows are not redistributed.
|
||
- **Quantized variants are evaluated separately**: each shipped GGUF carries
|
||
its own result table. The GGUF and bf16 harnesses differ in runtime and
|
||
stop handling, so score gaps are not attributed solely to quantization.
|
||
|
||
## Why two variants
|
||
|
||
Under fixed architecture and base checkpoint, modest differences in
|
||
fine-tuning composition produced large, cross-surface capability
|
||
tradeoffs (e.g., one build scores 96.8% on held-out carry addition while
|
||
the other scores 79.0% on held-out two-step word problems and 69.0% on
|
||
held-out function writing). Rather than presenting one checkpoint as
|
||
dominant, each variant ships with its measured envelope and a
|
||
recommendation for which profile
|
||
fits which pipeline. A checkpoint's profile is measured, never inferred.
|
||
|
||
## Shared limitations
|
||
|
||
Function-writing is evaluated for single Python functions only — neither
|
||
variant is a general code model (complete programs, executable SQL, and
|
||
bash scored near zero for both and are stated out of scope). Both variants
|
||
refused every designated refusal prompt, including real factual questions;
|
||
this targeted behavior is not evidence of general epistemic calibration.
|
||
Multi-step GSM-class reasoning is weak in every variant tested this cycle.
|
||
|
||
## Evidence disclosure
|
||
|
||
The public folders contain aggregate result tables, deterministic
|
||
representative examples from ConeML-owned instruments, and SHA-256
|
||
commitments to the complete row-level evidence. Full proprietary probe
|
||
rows are retained by ConeML rather than distributed. This preserves the
|
||
measurement instruments while allowing later disclosure to be checked
|
||
against the evidence frozen for this release.
|