141 lines
7.2 KiB
Markdown
141 lines
7.2 KiB
Markdown
# ConeML 810M peer diagnostic and resource context
|
||
|
||
Date: 2026-07-30
|
||
|
||
This is an interface-specific diagnostic screen, not a neutral ranking of
|
||
general model capability. Each instruction-tuned model was evaluated
|
||
through its own instruction interface. All models used greedy decoding
|
||
with repetition penalty 1.15. Arithmetic and function-writing used
|
||
matched short-answer generation budgets.
|
||
|
||
The task families match ConeML's supervised training surfaces. That gives
|
||
the comparison practical value for locating the release's envelope, but
|
||
it also means these results must not be generalized to unrelated tasks.
|
||
Standard GSM8K and HumanEval results remain separately disclosed in the
|
||
model cards.
|
||
|
||
## Instruct-model results
|
||
|
||
| measured surface | ConeML 810M Alpha | ConeML 810M Alpha-Arithmetic | Qwen3.5 0.8B | Qwen3 0.6B | Llama 3.2 1B Instruct | TinyLlama Chat | SmolLM2 1.7B Instruct |
|
||
|---|---:|---:|---:|---:|---:|---:|---:|
|
||
| mixed arithmetic screen (n=585) | 421 (72.0%) | **442 (75.6%)** | 154 (26.3%) | 223 (38.1%) | 343 (58.6%) | 113 (19.3%) | 380 (65.0%) |
|
||
| four core arithmetic lanes (n=225) | 177 (78.7%) | **221 (98.2%)** | 150 (66.7%) | 208 (92.4%) | **221 (98.2%)** | 92 (40.9%) | **221 (98.2%)** |
|
||
| executed single functions (n=100) | 83 | 35 | 93 | **98** | 79 | 51 | 96 |
|
||
| transitive names, depth 1/3/5 (n=32 each) | 23/15/15 | 26/17/16 | 23/22/10 | 18/10/9 | 16/7/5 | 8/18/22 | 16/6/5 |
|
||
| transitive entities, depth 1/3/5 (n=32 each) | 15/11/10 | 16/11/12 | 18/16/14 | 14/17/11 | 0/6/7 | 14/15/11 | 22/21/10 |
|
||
| designated-refusal prompts (n=17) | **13** | 11 | 0 | 1 | 1 | 0 | 1 |
|
||
| over-refusals on contrasts (n=5) | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
||
|
||
The honest result is mixed:
|
||
|
||
- At matched short-answer budgets, the ConeML pair led this post-trained
|
||
peer group on the broader arithmetic screen.
|
||
- On the four core arithmetic lanes, the Arithmetic variant tied Llama
|
||
3.2 Instruct and SmolLM2 at this reduced sample size. Its separate
|
||
full-size internal result is 1,093/1,116 (97.9%).
|
||
- ConeML 810M Alpha exceeded Llama 3.2 Instruct and TinyLlama Chat on the
|
||
executed function screen, but Qwen3.5, Qwen3, and SmolLM2 scored higher.
|
||
- On name-chain transitive selection, ConeML 810M Alpha-Arithmetic
|
||
recorded 26/32, 17/32,
|
||
and 16/32 at depths 1, 3, and 5. It exceeded Qwen3 and Llama 3.2 at all
|
||
three shown depths and exceeded Qwen3.5 at depths 1 and 5; Qwen3.5 led
|
||
it at depth 3, and TinyLlama led the group at depth 5.
|
||
- Entity-chain transfer was mixed: ConeML 810M Alpha recorded 15/11/10
|
||
and ConeML 810M Alpha-Arithmetic 16/11/12 at depths 1/3/5, while different peers led each
|
||
depth. Chance is 1/(depth+1). The non-monotonic peer rows reinforce
|
||
that this is an exact-selection surface test, not proof of general
|
||
reasoning depth.
|
||
- The refusal row measures a trained response policy on designated
|
||
prompts, not factual correctness or general epistemic calibration.
|
||
|
||
The peer-harness refusal results above differ from the dedicated ConeML
|
||
probe's 17/17 for each release because the generation loops differ. Both
|
||
measurements are reported rather than merged.
|
||
|
||
## Qwen3.5 thinking-mode sensitivity
|
||
|
||
Qwen3.5 0.8B was also evaluated with thinking enabled on the same 585
|
||
arithmetic items:
|
||
|
||
| metric | Qwen3.5 thinking | Qwen3.5 short-answer | ConeML 810M Alpha-Arithmetic short-answer |
|
||
|---|---:|---:|---:|
|
||
| accuracy | 451/585 (77.1%) | 154/585 (26.3%) | 442/585 (75.6%) |
|
||
| generated tokens per item | mean 844; median 699 | at most 48 | at most 48 |
|
||
| p95 / generation cap | 1,536 / 1,536 | 48 / 48 | 48 / 48 |
|
||
| truncation rate | 138/585 (23.6%) | approximately 0% | approximately 0% |
|
||
| mean wall time per item | 2.43 s | approximately 0.1 s | approximately 0.1 s |
|
||
| correct answers per 1,000 generated tokens | 0.91 | 5.5 | approximately 15.7 or higher |
|
||
|
||
Thinking mode recovered Qwen3.5 to statistical parity with ConeML 810M
|
||
Alpha-Arithmetic on this screen, while using at least 17.6 times the
|
||
per-answer generation budget and approximately 24 times the measured wall
|
||
time. Wall-time ratios are specific to the recorded hardware, batching,
|
||
and implementation. This is an inference-cost comparison, not a claim
|
||
that thinking mode is intrinsically inferior.
|
||
|
||
## Against-interest base result
|
||
|
||
On the same 585 arithmetic items, through a task frame native to neither
|
||
base model, Qwen3.5 0.8B Base scored 492/585 (84.1%), while ConeML base
|
||
checkpoint 188 scored 156/585 (26.7%). This result is included because it
|
||
prevents an absolute-superiority reading: high-exposure base models may
|
||
already contain strong task-formatted behavior, and post-training can
|
||
move capability between output surfaces.
|
||
|
||
## Training-resource context
|
||
|
||
ConeML's selected base consumed approximately 12.32B token positions:
|
||
15.2 tokens per parameter and approximately 5.99e19 training FLOPs under
|
||
the `6 × parameters × tokens` convention.
|
||
|
||
| model family | disclosed pretraining tokens | approximate tokens/parameter | approximate training FLOPs vs ConeML |
|
||
|---|---:|---:|---:|
|
||
| ConeML 810M | 12.32B | 15.2 | 1× |
|
||
| TinyLlama 1.1B | 3T | 2,727 | 330× |
|
||
| Llama 3.2 1B | up to 9T | 7,258 | 1,118×, plus distillation |
|
||
| SmolLM2 1.7B | 11T | 6,471 | 1,873× |
|
||
| Qwen3 0.6B | 36T | 60,000 | 2,164× |
|
||
| Qwen3.5 0.8B | not disclosed | not stated | not stated |
|
||
|
||
These are estimated pretraining FLOP ratios, not historical electricity
|
||
or monetary costs for the peer models.
|
||
|
||
ConeML pretraining took approximately 11 days on one local RTX 5090. At
|
||
an explicitly assumed average wall draw of 0.70 kW, that corresponds to
|
||
184.8 kWh. Applying an assumed Swiss residential tariff range of
|
||
CHF 0.14–0.30/kWh gives approximately CHF 26–55 of marginal pretraining
|
||
electricity.
|
||
|
||
| ConeML pretraining unit | estimate |
|
||
|---|---:|
|
||
| average throughput | 12,963 token positions/s |
|
||
| wall time per billion token positions | 21.4 h |
|
||
| energy per billion token positions | 15.0 kWh |
|
||
| energy per million token positions | 15 Wh |
|
||
| estimated wall energy per token position | 0.054 J |
|
||
| marginal electricity per billion token positions | CHF 2.10–4.50 |
|
||
|
||
The energy and cost figures are estimates derived from the stated power
|
||
and tariff assumptions, not meter readings. They exclude hardware,
|
||
depreciation, labor, supervised fine-tuning, evaluation, conversion,
|
||
datacenter PUE, and carbon intensity. No emissions claim is made.
|
||
|
||
## Evaluated revisions
|
||
|
||
- `Qwen/Qwen3.5-0.8B@2fc06364715b967f1860aea9cf38778875588b17`
|
||
- `Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca`
|
||
- `unsloth/Llama-3.2-1B-Instruct@5a8abab4a5d6f164389b1079fb721cfab8d7126c`
|
||
- `TinyLlama/TinyLlama-1.1B-Chat-v1.0@fe8a4ea1ffedaf415f4da2f062534de366a451e6`
|
||
- `HuggingFaceTB/SmolLM2-1.7B-Instruct@31b70e2e869a7173562077fd711b654946d38674`
|
||
|
||
The exact aggregates are in `peer-comparison-summary.json`. Complete
|
||
generation rows are retained privately; their frozen SHA-256 commitments
|
||
are published in `PEER_EVIDENCE_SHA256SUMS.txt`.
|
||
|
||
## Primary resource disclosures
|
||
|
||
- Qwen3 pretraining: [Qwen3 release post](https://qwenlm.github.io/blog/qwen3/)
|
||
- Llama 3.2 token count and distillation: [Meta Llama 3.2 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md)
|
||
- TinyLlama token count: [TinyLlama model card](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)
|
||
- SmolLM2 token count: [Hugging Face SmolLM2 model card](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B)
|