初始化项目，由ModelHub XC社区提供模型
Model: ConeML/coneml-810m-alpha-arithmetic Source: Original Platform
This commit is contained in:
ModelHub XC
59
eval/EVALUATION_METHODOLOGY.md
Normal file
59
eval/EVALUATION_METHODOLOGY.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# ConeML 810M — evaluation methodology and variant lineage (public record)
|
||||
|
||||
Two released variants share one from-scratch 810M base checkpoint and
|
||||
differ only in supervised fine-tuning; both were evaluated with the same
|
||||
bf16 instruments.
|
||||
|
||||
- `coneml-810m-alpha` — function-writing / mixed-arithmetic profile
|
||||
- `coneml-810m-alpha-arithmetic` — carry/borrow arithmetic variant
|
||||
|
||||
## Methodology
|
||||
|
||||
- **ConeML internal certification**: large-n (300–500 per task family)
|
||||
batteries of unseen instances in the trained prompt formats, greedy
|
||||
decoding, repetition
|
||||
penalty 1.15. Code tasks are scored by EXECUTING the generated function
|
||||
against held-out tests; SQL by executing against a live sqlite database;
|
||||
scripts by captured stdout. No pattern-match scoring for anything
|
||||
execution can decide.
|
||||
- **Behavior probes**: held-out designated-refusal prompts plus in-scope
|
||||
contrast pairs in matching wording; one separately rerun, scripted
|
||||
eight-turn conversation; and an everyday-reasoning screen with manual
|
||||
adjudication. Aggregate results and deterministic representative
|
||||
ConeML-owned rows are public; complete rows are retained privately.
|
||||
- **Public benchmarks**: GSM8K (first 200, zero-shot, strict final-number
|
||||
match) and HumanEval (pass@1, tests executed), identical harness across
|
||||
variants and the pretrained base. Aggregate results are public;
|
||||
benchmark rows are not redistributed.
|
||||
- **Quantized variants are evaluated separately**: each shipped GGUF carries
|
||||
its own result table. The GGUF and bf16 harnesses differ in runtime and
|
||||
stop handling, so score gaps are not attributed solely to quantization.
|
||||
|
||||
## Why two variants
|
||||
|
||||
Under fixed architecture and base checkpoint, modest differences in
|
||||
fine-tuning composition produced large, cross-surface capability
|
||||
tradeoffs (e.g., one build scores 96.8% on held-out carry addition while
|
||||
the other scores 79.0% on held-out two-step word problems and 69.0% on
|
||||
held-out function writing). Rather than presenting one checkpoint as
|
||||
dominant, each variant ships with its measured envelope and a
|
||||
recommendation for which profile
|
||||
fits which pipeline. A checkpoint's profile is measured, never inferred.
|
||||
|
||||
## Shared limitations
|
||||
|
||||
Function-writing is evaluated for single Python functions only — neither
|
||||
variant is a general code model (complete programs, executable SQL, and
|
||||
bash scored near zero for both and are stated out of scope). Both variants
|
||||
refused every designated refusal prompt, including real factual questions;
|
||||
this targeted behavior is not evidence of general epistemic calibration.
|
||||
Multi-step GSM-class reasoning is weak in every variant tested this cycle.
|
||||
|
||||
## Evidence disclosure
|
||||
|
||||
The public folders contain aggregate result tables, deterministic
|
||||
representative examples from ConeML-owned instruments, and SHA-256
|
||||
commitments to the complete row-level evidence. Full proprietary probe
|
||||
rows are retained by ConeML rather than distributed. This preserves the
|
||||
measurement instruments while allowing later disclosure to be checked
|
||||
against the evidence frozen for this release.
|
||||
140
eval/PEER_COMPARISON.md
Normal file
140
eval/PEER_COMPARISON.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# ConeML 810M peer diagnostic and resource context
|
||||
|
||||
Date: 2026-07-30
|
||||
|
||||
This is an interface-specific diagnostic screen, not a neutral ranking of
|
||||
general model capability. Each instruction-tuned model was evaluated
|
||||
through its own instruction interface. All models used greedy decoding
|
||||
with repetition penalty 1.15. Arithmetic and function-writing used
|
||||
matched short-answer generation budgets.
|
||||
|
||||
The task families match ConeML's supervised training surfaces. That gives
|
||||
the comparison practical value for locating the release's envelope, but
|
||||
it also means these results must not be generalized to unrelated tasks.
|
||||
Standard GSM8K and HumanEval results remain separately disclosed in the
|
||||
model cards.
|
||||
|
||||
## Instruct-model results
|
||||
|
||||
| measured surface | ConeML 810M Alpha | ConeML 810M Alpha-Arithmetic | Qwen3.5 0.8B | Qwen3 0.6B | Llama 3.2 1B Instruct | TinyLlama Chat | SmolLM2 1.7B Instruct |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|
|
||||
| mixed arithmetic screen (n=585) | 421 (72.0%) | **442 (75.6%)** | 154 (26.3%) | 223 (38.1%) | 343 (58.6%) | 113 (19.3%) | 380 (65.0%) |
|
||||
| four core arithmetic lanes (n=225) | 177 (78.7%) | **221 (98.2%)** | 150 (66.7%) | 208 (92.4%) | **221 (98.2%)** | 92 (40.9%) | **221 (98.2%)** |
|
||||
| executed single functions (n=100) | 83 | 35 | 93 | **98** | 79 | 51 | 96 |
|
||||
| transitive names, depth 1/3/5 (n=32 each) | 23/15/15 | 26/17/16 | 23/22/10 | 18/10/9 | 16/7/5 | 8/18/22 | 16/6/5 |
|
||||
| transitive entities, depth 1/3/5 (n=32 each) | 15/11/10 | 16/11/12 | 18/16/14 | 14/17/11 | 0/6/7 | 14/15/11 | 22/21/10 |
|
||||
| designated-refusal prompts (n=17) | **13** | 11 | 0 | 1 | 1 | 0 | 1 |
|
||||
| over-refusals on contrasts (n=5) | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
||||
|
||||
The honest result is mixed:
|
||||
|
||||
- At matched short-answer budgets, the ConeML pair led this post-trained
|
||||
peer group on the broader arithmetic screen.
|
||||
- On the four core arithmetic lanes, the Arithmetic variant tied Llama
|
||||
3.2 Instruct and SmolLM2 at this reduced sample size. Its separate
|
||||
full-size internal result is 1,093/1,116 (97.9%).
|
||||
- ConeML 810M Alpha exceeded Llama 3.2 Instruct and TinyLlama Chat on the
|
||||
executed function screen, but Qwen3.5, Qwen3, and SmolLM2 scored higher.
|
||||
- On name-chain transitive selection, ConeML 810M Alpha-Arithmetic
|
||||
recorded 26/32, 17/32,
|
||||
and 16/32 at depths 1, 3, and 5. It exceeded Qwen3 and Llama 3.2 at all
|
||||
three shown depths and exceeded Qwen3.5 at depths 1 and 5; Qwen3.5 led
|
||||
it at depth 3, and TinyLlama led the group at depth 5.
|
||||
- Entity-chain transfer was mixed: ConeML 810M Alpha recorded 15/11/10
|
||||
and ConeML 810M Alpha-Arithmetic 16/11/12 at depths 1/3/5, while different peers led each
|
||||
depth. Chance is 1/(depth+1). The non-monotonic peer rows reinforce
|
||||
that this is an exact-selection surface test, not proof of general
|
||||
reasoning depth.
|
||||
- The refusal row measures a trained response policy on designated
|
||||
prompts, not factual correctness or general epistemic calibration.
|
||||
|
||||
The peer-harness refusal results above differ from the dedicated ConeML
|
||||
probe's 17/17 for each release because the generation loops differ. Both
|
||||
measurements are reported rather than merged.
|
||||
|
||||
## Qwen3.5 thinking-mode sensitivity
|
||||
|
||||
Qwen3.5 0.8B was also evaluated with thinking enabled on the same 585
|
||||
arithmetic items:
|
||||
|
||||
| metric | Qwen3.5 thinking | Qwen3.5 short-answer | ConeML 810M Alpha-Arithmetic short-answer |
|
||||
|---|---:|---:|---:|
|
||||
| accuracy | 451/585 (77.1%) | 154/585 (26.3%) | 442/585 (75.6%) |
|
||||
| generated tokens per item | mean 844; median 699 | at most 48 | at most 48 |
|
||||
| p95 / generation cap | 1,536 / 1,536 | 48 / 48 | 48 / 48 |
|
||||
| truncation rate | 138/585 (23.6%) | approximately 0% | approximately 0% |
|
||||
| mean wall time per item | 2.43 s | approximately 0.1 s | approximately 0.1 s |
|
||||
| correct answers per 1,000 generated tokens | 0.91 | 5.5 | approximately 15.7 or higher |
|
||||
|
||||
Thinking mode recovered Qwen3.5 to statistical parity with ConeML 810M
|
||||
Alpha-Arithmetic on this screen, while using at least 17.6 times the
|
||||
per-answer generation budget and approximately 24 times the measured wall
|
||||
time. Wall-time ratios are specific to the recorded hardware, batching,
|
||||
and implementation. This is an inference-cost comparison, not a claim
|
||||
that thinking mode is intrinsically inferior.
|
||||
|
||||
## Against-interest base result
|
||||
|
||||
On the same 585 arithmetic items, through a task frame native to neither
|
||||
base model, Qwen3.5 0.8B Base scored 492/585 (84.1%), while ConeML base
|
||||
checkpoint 188 scored 156/585 (26.7%). This result is included because it
|
||||
prevents an absolute-superiority reading: high-exposure base models may
|
||||
already contain strong task-formatted behavior, and post-training can
|
||||
move capability between output surfaces.
|
||||
|
||||
## Training-resource context
|
||||
|
||||
ConeML's selected base consumed approximately 12.32B token positions:
|
||||
15.2 tokens per parameter and approximately 5.99e19 training FLOPs under
|
||||
the `6 × parameters × tokens` convention.
|
||||
|
||||
| model family | disclosed pretraining tokens | approximate tokens/parameter | approximate training FLOPs vs ConeML |
|
||||
|---|---:|---:|---:|
|
||||
| ConeML 810M | 12.32B | 15.2 | 1× |
|
||||
| TinyLlama 1.1B | 3T | 2,727 | 330× |
|
||||
| Llama 3.2 1B | up to 9T | 7,258 | 1,118×, plus distillation |
|
||||
| SmolLM2 1.7B | 11T | 6,471 | 1,873× |
|
||||
| Qwen3 0.6B | 36T | 60,000 | 2,164× |
|
||||
| Qwen3.5 0.8B | not disclosed | not stated | not stated |
|
||||
|
||||
These are estimated pretraining FLOP ratios, not historical electricity
|
||||
or monetary costs for the peer models.
|
||||
|
||||
ConeML pretraining took approximately 11 days on one local RTX 5090. At
|
||||
an explicitly assumed average wall draw of 0.70 kW, that corresponds to
|
||||
184.8 kWh. Applying an assumed Swiss residential tariff range of
|
||||
CHF 0.14–0.30/kWh gives approximately CHF 26–55 of marginal pretraining
|
||||
electricity.
|
||||
|
||||
| ConeML pretraining unit | estimate |
|
||||
|---|---:|
|
||||
| average throughput | 12,963 token positions/s |
|
||||
| wall time per billion token positions | 21.4 h |
|
||||
| energy per billion token positions | 15.0 kWh |
|
||||
| energy per million token positions | 15 Wh |
|
||||
| estimated wall energy per token position | 0.054 J |
|
||||
| marginal electricity per billion token positions | CHF 2.10–4.50 |
|
||||
|
||||
The energy and cost figures are estimates derived from the stated power
|
||||
and tariff assumptions, not meter readings. They exclude hardware,
|
||||
depreciation, labor, supervised fine-tuning, evaluation, conversion,
|
||||
datacenter PUE, and carbon intensity. No emissions claim is made.
|
||||
|
||||
## Evaluated revisions
|
||||
|
||||
- `Qwen/Qwen3.5-0.8B@2fc06364715b967f1860aea9cf38778875588b17`
|
||||
- `Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca`
|
||||
- `unsloth/Llama-3.2-1B-Instruct@5a8abab4a5d6f164389b1079fb721cfab8d7126c`
|
||||
- `TinyLlama/TinyLlama-1.1B-Chat-v1.0@fe8a4ea1ffedaf415f4da2f062534de366a451e6`
|
||||
- `HuggingFaceTB/SmolLM2-1.7B-Instruct@31b70e2e869a7173562077fd711b654946d38674`
|
||||
|
||||
The exact aggregates are in `peer-comparison-summary.json`. Complete
|
||||
generation rows are retained privately; their frozen SHA-256 commitments
|
||||
are published in `PEER_EVIDENCE_SHA256SUMS.txt`.
|
||||
|
||||
## Primary resource disclosures
|
||||
|
||||
- Qwen3 pretraining: [Qwen3 release post](https://qwenlm.github.io/blog/qwen3/)
|
||||
- Llama 3.2 token count and distillation: [Meta Llama 3.2 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md)
|
||||
- TinyLlama token count: [TinyLlama model card](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)
|
||||
- SmolLM2 token count: [Hugging Face SmolLM2 model card](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B)
|
||||
35
eval/PEER_EVIDENCE_SHA256SUMS.txt
Normal file
35
eval/PEER_EVIDENCE_SHA256SUMS.txt
Normal file
@@ -0,0 +1,35 @@
|
||||
96a805aed179f77d9f1959216f9a39e8bf8d50dea5465eeea1eb24e96502a6af SOURCE_PEER_COMPARISON_V2.md
|
||||
a5d6cd4d7a2643b316cefb1dcb6f43296bb69da4064dbc3b3ab4a123bf36b2b1 base-reference/coneml-base188.json
|
||||
83643ce110655e708d50f4eb3ee29ab08ab8e003768eb5b5da9a3e545bc9b8ef base-reference/coneml-base188.rows.jsonl
|
||||
ca387426c55ac72d50041e2f082bd4cdc747eedb51deca7929cf7a4acea2c5b6 base-reference/coneml-native-transitive.json
|
||||
43ef584d37c6a276b190aeb741597b54bfebbf7bbd9c37df5d82a2e15a8ee2d1 base-reference/qwen35-0.8b-base.json
|
||||
25de33180d4073777d29871882b57f1556733e9a8c1e10eba14a8cac062b5c35 base-reference/qwen35-0.8b-base.rows.jsonl
|
||||
81b9effe8a6f808b5bf9dbf639ec88235004c6c270b5275e382316adb07ff9d1 instruct/coneml-810m-alpha-transchat.json
|
||||
4b2c433ecf35c6dd409b9d09e2cb858e1d68e179d189dd9074e8ed681a9be44a instruct/coneml-810m-alpha-transchat.rows.jsonl
|
||||
2003c73a54602f3a30980d6c7f85feef39b63fd11cef9fc20a90044d3569ba77 instruct/coneml-810m-alpha.json
|
||||
8da107545bd6717c7795ff4e827698e5e269d451f2390f24dccc516fc390ab44 instruct/coneml-810m-alpha.rows.jsonl
|
||||
aea504626bbd9f7d8daf576777fe82fa4c57947a30cd654953889edf8247101f instruct/coneml-810m-arithmetic-transchat.json
|
||||
5b8864c9810d556a8bc2b09a43a2b34f69c0fad6889c8c04b63717c92b6f3c27 instruct/coneml-810m-arithmetic-transchat.rows.jsonl
|
||||
77d514c14e14557c4ac30e8a8c3f4c5887f0db0dff39b8ea14baea164327d027 instruct/coneml-810m-arithmetic.json
|
||||
a60ed3e533c0c4a77b34a21f80b8df8de85e3df074f3cacdc3bbd4b5e0741cce instruct/coneml-810m-arithmetic.rows.jsonl
|
||||
aa35b93f46778ffb22351ac411f2fc04e645775e416b5a9423f925bb6d6b1609 instruct/llama32-1b-instruct-transchat.json
|
||||
ee514c35d86f27172a004325b26da915f869164adae59edf8d95b01d958e683f instruct/llama32-1b-instruct-transchat.rows.jsonl
|
||||
f4553ef34844e55364a87f631fac03a3545acbb14cd12dcf182f16151b5e5f8a instruct/llama32-1b-instruct.json
|
||||
d5c0fd8b00b50754ac7dede96081afeaf3143795e005134cf5543483859340bf instruct/llama32-1b-instruct.rows.jsonl
|
||||
a61b56681810410dfc3a8d8e758b8ccd11d01ebcce2d963d6bdbca4644c2cee9 instruct/qwen3-0.6b-transchat.json
|
||||
f7c763441edc0d5e37a34902a3cc4efcacc6a35b47056adb9356879a4407cb31 instruct/qwen3-0.6b-transchat.rows.jsonl
|
||||
3f34342d79095a5754512ebf57192fdf98a16b364a250712993ac1da0b65e252 instruct/qwen3-0.6b.json
|
||||
05847f4187651780008331a85c9cd5fa84d3b636320c9a809fb3c307f1a230d6 instruct/qwen3-0.6b.rows.jsonl
|
||||
21984e00a4a6da3d2887138ede25c03e9007fee0be41de462ef58dc58db89554 instruct/qwen35-0.8b-transchat.json
|
||||
48a53b6bf3829c207953b1075160df9f6702b1fc2162426577aa69530856e555 instruct/qwen35-0.8b-transchat.rows.jsonl
|
||||
fa11fbf67c047ac541ecd38571867f9fe68bfb8aec12b84f2addda8a6d561642 instruct/qwen35-0.8b.json
|
||||
74a6debf6ae55db9895f556e654b6905fde871358a94c9c4a2730a418b0e2608 instruct/qwen35-0.8b.rows.jsonl
|
||||
72598668bfce72497ac67a4d83d1cad7538c63ec933e8a1d1198876d85dc8a67 instruct/qwen35-thinking-sensitivity.json
|
||||
8917677a4c9d29124c5ade229f7080bbd0ccb83471c3d7d39a6f958224596a8f instruct/smollm2-1.7b-instruct-transchat.json
|
||||
deb9eb805475e4ad2d158283b469830138fdf42ee0c9a903bf3611b43a7869ff instruct/smollm2-1.7b-instruct-transchat.rows.jsonl
|
||||
94a3440cdaae14eb27beaf36a1a0103dab4a6d292bc382981f0514a82a93ed0f instruct/smollm2-1.7b-instruct.json
|
||||
1b06370ad7b940ed05b59ffe4acd19035341cfe078ed7b01d59d168abc8d0c71 instruct/smollm2-1.7b-instruct.rows.jsonl
|
||||
bdd7cf2867643439e90a3e04cb209e6130839af9422e492bcd2eb1c2cba4556e instruct/tinyllama-chat-transchat.json
|
||||
472aad0c169c4f943fbc9a8367d65fa2d5c41acbbee66d099e16a286f8676d1c instruct/tinyllama-chat-transchat.rows.jsonl
|
||||
3a7c2e7ebcaa78b4b7d9b6d84a5b90a495f260f97287bad3282d54fc84e0c009 instruct/tinyllama-chat.json
|
||||
5926a31e926279f8d9459d19cbbe9730a06071bb8b4029c2f9174804ab447943 instruct/tinyllama-chat.rows.jsonl
|
||||
10
eval/PRIVATE_EVIDENCE_SHA256SUMS.txt
Normal file
10
eval/PRIVATE_EVIDENCE_SHA256SUMS.txt
Normal file
@@ -0,0 +1,10 @@
|
||||
532b8cd9c9f7eb7cd047aac26dfe6a99880503a7b09bf79314287c28cb908ce4 base188-public-benchmarks.json
|
||||
9ae0002c88cd5972d21fe119734421f03f8ffcaa412002476ae5b0244bc302b7 basic-code.json
|
||||
83ff212bfd15d342856cf1ececf46648d458b1a0010afc1538b3a87b7ef1b86b certification-v1.json
|
||||
bed82d5fefc647f91d53cd22376fbb3b598ea417e317e698629bc82d07b6479c code-cert.json
|
||||
3b263314c8690fb912a03fc87b5d81dd38e499a6f3ef4353dee124239572a826 conversation.log
|
||||
ebca66bd78a552bf8e70498f5b38a778e27894afb2f47c122da29195d85dba3e conversation-v2.json
|
||||
e7c6a5a5e7b7cb8c35cba5545cc56e79e01571e550a5958cad0ad3e9a0d1b9d6 everyday-manual-adjudication.json
|
||||
3b698faa42fe5b556abcb7ba4ac39866a94ffd2413dcd5987251e9c5b46b1b91 public-benchmarks.json
|
||||
3e28a81a3218e97503a4f572c5dc8cedd4943b02bf3de046614661f0d88bb247 raw-probe.jsonl
|
||||
66f34c9e24e01a3e26db9faaf3606e510eb42639cd32169e500e17d1a2d53b38 refusal-probe.json
|
||||
55
eval/certification-Q8_0.json
Normal file
55
eval/certification-Q8_0.json
Normal file
@@ -0,0 +1,55 @@
|
||||
{
|
||||
"by_category": {
|
||||
"add-1d": {
|
||||
"correct": 70,
|
||||
"n": 71,
|
||||
"acc": 0.9859
|
||||
},
|
||||
"add-2d-carry": {
|
||||
"correct": 463,
|
||||
"n": 500,
|
||||
"acc": 0.926
|
||||
},
|
||||
"comparison": {
|
||||
"correct": 363,
|
||||
"n": 500,
|
||||
"acc": 0.726
|
||||
},
|
||||
"missing-addend": {
|
||||
"correct": 337,
|
||||
"n": 500,
|
||||
"acc": 0.674
|
||||
},
|
||||
"missing-factor": {
|
||||
"correct": 92,
|
||||
"n": 500,
|
||||
"acc": 0.184
|
||||
},
|
||||
"mul-1d": {
|
||||
"correct": 45,
|
||||
"n": 45,
|
||||
"acc": 1.0
|
||||
},
|
||||
"mul-2dx1d": {
|
||||
"correct": 398,
|
||||
"n": 500,
|
||||
"acc": 0.796
|
||||
},
|
||||
"sub-borrow": {
|
||||
"correct": 493,
|
||||
"n": 500,
|
||||
"acc": 0.986
|
||||
},
|
||||
"two-step": {
|
||||
"correct": 163,
|
||||
"n": 500,
|
||||
"acc": 0.326
|
||||
},
|
||||
"which-bigger": {
|
||||
"correct": 277,
|
||||
"n": 300,
|
||||
"acc": 0.9233
|
||||
}
|
||||
},
|
||||
"runtime": "llama.cpp Q8_0 aggregate summary"
|
||||
}
|
||||
25
eval/focused-summary.json
Normal file
25
eval/focused-summary.json
Normal file
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"ckpt": "ConeML/coneml-810m-alpha-arithmetic",
|
||||
"math_200": 138,
|
||||
"everyday_20_auto": 6,
|
||||
"by_category": {
|
||||
"everyday::causal-evidence": "0/3",
|
||||
"everyday::everyday-arithmetic": "2/2",
|
||||
"everyday::money-change": "0/3",
|
||||
"everyday::object-choice": "2/3",
|
||||
"everyday::practical-judgment": "1/2",
|
||||
"everyday::quantity-judgment": "0/2",
|
||||
"everyday::temporal-order": "0/2",
|
||||
"everyday::time-margin": "1/3",
|
||||
"math::add-1d": "20/20",
|
||||
"math::add-2d-carry": "30/30",
|
||||
"math::missing-addend": "18/20",
|
||||
"math::mul": "20/20",
|
||||
"math::sub-2d-borrow": "29/30",
|
||||
"math::transitive": "0/20",
|
||||
"math::word-2step": "2/40",
|
||||
"math::word-compare": "19/20"
|
||||
},
|
||||
"math_200_corrected": 154,
|
||||
"correction": "transitive items scored by expected-name match on first line; original scorer extracted numbers only and zeroed all transitive rows"
|
||||
}
|
||||
140
eval/peer-comparison-summary.json
Normal file
140
eval/peer-comparison-summary.json
Normal file
@@ -0,0 +1,140 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"date": "2026-07-30",
|
||||
"scope": "Interface-specific diagnostic screen, not a neutral ranking of general model capability. Each instruct model used its native instruction interface. Arithmetic and function-writing used matched short-answer generation budgets.",
|
||||
"decoding": {
|
||||
"strategy": "greedy",
|
||||
"repetition_penalty": 1.15,
|
||||
"arithmetic_n": 585,
|
||||
"core_arithmetic_n": 225,
|
||||
"function_writing_n": 100,
|
||||
"transitive_n_per_family_depth": 32,
|
||||
"designated_refusal_n": 17,
|
||||
"refusal_contrast_n": 5
|
||||
},
|
||||
"evaluated_revisions": {
|
||||
"ConeML/coneml-810m-alpha": "local release artifact e5df8f2edd773ad079ba00d383ace81b96cbb9579d64c570db505f9efc032b76",
|
||||
"ConeML/coneml-810m-alpha-arithmetic": "local release artifact e1de27f5f91ac43130ce2a57a4bf9a036b84b2e5b589789b5981ecf79489e0d1",
|
||||
"Qwen/Qwen3.5-0.8B": "2fc06364715b967f1860aea9cf38778875588b17",
|
||||
"Qwen/Qwen3-0.6B": "c1899de289a04d12100db370d81485cdf75e47ca",
|
||||
"unsloth/Llama-3.2-1B-Instruct": "5a8abab4a5d6f164389b1079fb721cfab8d7126c",
|
||||
"TinyLlama/TinyLlama-1.1B-Chat-v1.0": "fe8a4ea1ffedaf415f4da2f062534de366a451e6",
|
||||
"HuggingFaceTB/SmolLM2-1.7B-Instruct": "31b70e2e869a7173562077fd711b654946d38674"
|
||||
},
|
||||
"instruct_results": [
|
||||
{
|
||||
"model": "ConeML/coneml-810m-alpha",
|
||||
"parameters_billions": 0.8101,
|
||||
"arithmetic_mixed": {"correct": 421, "n": 585, "accuracy": 0.7197},
|
||||
"arithmetic_core_four_lanes": {"correct": 177, "n": 225, "accuracy": 0.7867},
|
||||
"executed_functions": {"correct": 83, "n": 100, "accuracy": 0.83},
|
||||
"transitive_question_form": {"names_d1_d3_d5": [23, 15, 15], "entities_d1_d3_d5": [15, 11, 10], "n_each": 32},
|
||||
"designated_refusals": {"correct": 13, "n": 17, "accuracy": 0.7647},
|
||||
"over_refusals": {"count": 0, "n": 5}
|
||||
},
|
||||
{
|
||||
"model": "ConeML/coneml-810m-alpha-arithmetic",
|
||||
"parameters_billions": 0.8101,
|
||||
"arithmetic_mixed": {"correct": 442, "n": 585, "accuracy": 0.7556},
|
||||
"arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
|
||||
"executed_functions": {"correct": 35, "n": 100, "accuracy": 0.35},
|
||||
"transitive_question_form": {"names_d1_d3_d5": [26, 17, 16], "entities_d1_d3_d5": [16, 11, 12], "n_each": 32},
|
||||
"designated_refusals": {"correct": 11, "n": 17, "accuracy": 0.6471},
|
||||
"over_refusals": {"count": 0, "n": 5}
|
||||
},
|
||||
{
|
||||
"model": "Qwen/Qwen3.5-0.8B",
|
||||
"parameters_billions": 0.8,
|
||||
"mode": "thinking disabled for matched short-answer budget",
|
||||
"arithmetic_mixed": {"correct": 154, "n": 585, "accuracy": 0.2632},
|
||||
"arithmetic_core_four_lanes": {"correct": 150, "n": 225, "accuracy": 0.6667},
|
||||
"executed_functions": {"correct": 93, "n": 100, "accuracy": 0.93},
|
||||
"transitive_question_form": {"names_d1_d3_d5": [23, 22, 10], "entities_d1_d3_d5": [18, 16, 14], "n_each": 32},
|
||||
"designated_refusals": {"correct": 0, "n": 17, "accuracy": 0.0},
|
||||
"over_refusals": {"count": 0, "n": 5}
|
||||
},
|
||||
{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"parameters_billions": 0.6,
|
||||
"arithmetic_mixed": {"correct": 223, "n": 585, "accuracy": 0.3812},
|
||||
"arithmetic_core_four_lanes": {"correct": 208, "n": 225, "accuracy": 0.9244},
|
||||
"executed_functions": {"correct": 98, "n": 100, "accuracy": 0.98},
|
||||
"transitive_question_form": {"names_d1_d3_d5": [18, 10, 9], "entities_d1_d3_d5": [14, 17, 11], "n_each": 32},
|
||||
"designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
|
||||
"over_refusals": {"count": 0, "n": 5}
|
||||
},
|
||||
{
|
||||
"model": "unsloth/Llama-3.2-1B-Instruct",
|
||||
"parameters_billions": 1.24,
|
||||
"arithmetic_mixed": {"correct": 343, "n": 585, "accuracy": 0.5863},
|
||||
"arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
|
||||
"executed_functions": {"correct": 79, "n": 100, "accuracy": 0.79},
|
||||
"transitive_question_form": {"names_d1_d3_d5": [16, 7, 5], "entities_d1_d3_d5": [0, 6, 7], "n_each": 32},
|
||||
"designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
|
||||
"over_refusals": {"count": 0, "n": 5}
|
||||
},
|
||||
{
|
||||
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
|
||||
"parameters_billions": 1.1,
|
||||
"arithmetic_mixed": {"correct": 113, "n": 585, "accuracy": 0.1932},
|
||||
"arithmetic_core_four_lanes": {"correct": 92, "n": 225, "accuracy": 0.4089},
|
||||
"executed_functions": {"correct": 51, "n": 100, "accuracy": 0.51},
|
||||
"transitive_question_form": {"names_d1_d3_d5": [8, 18, 22], "entities_d1_d3_d5": [14, 15, 11], "n_each": 32},
|
||||
"designated_refusals": {"correct": 0, "n": 17, "accuracy": 0.0},
|
||||
"over_refusals": {"count": 0, "n": 5}
|
||||
},
|
||||
{
|
||||
"model": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
|
||||
"parameters_billions": 1.7,
|
||||
"arithmetic_mixed": {"correct": 380, "n": 585, "accuracy": 0.6496},
|
||||
"arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
|
||||
"executed_functions": {"correct": 96, "n": 100, "accuracy": 0.96},
|
||||
"transitive_question_form": {"names_d1_d3_d5": [16, 6, 5], "entities_d1_d3_d5": [22, 21, 10], "n_each": 32},
|
||||
"designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
|
||||
"over_refusals": {"count": 0, "n": 5}
|
||||
}
|
||||
],
|
||||
"qwen35_thinking_sensitivity": {
|
||||
"model": "Qwen/Qwen3.5-0.8B",
|
||||
"mode": "thinking enabled",
|
||||
"arithmetic_mixed": {"correct": 451, "n": 585, "accuracy": 0.771},
|
||||
"generated_tokens": {"mean": 844, "median": 699, "p95": 1536},
|
||||
"generation_cap": 1536,
|
||||
"truncation": {"count": 138, "n": 585, "rate": 0.2359},
|
||||
"wall_seconds_per_item_mean": 2.43,
|
||||
"correct_per_1000_generated_tokens": 0.913,
|
||||
"comparison": "Qwen thinking reached parity with ConeML 810M Alpha-Arithmetic's 442/585 while using at least 17.6 times the generated-token budget per item. The recorded wall-time ratio was approximately 24 times. Runtime measurements are hardware- and implementation-specific."
|
||||
},
|
||||
"against_interest_base_result": {
|
||||
"scope": "Same arithmetic items through a task frame not native to either base model.",
|
||||
"ConeML_base_188": {"correct": 156, "n": 585, "accuracy": 0.2667},
|
||||
"Qwen3.5_0.8B_Base": {"correct": 492, "n": 585, "accuracy": 0.841},
|
||||
"interpretation": "The peer base result prevents an absolute-superiority interpretation and illustrates sensitivity to training exposure and interface."
|
||||
},
|
||||
"training_resource_context": {
|
||||
"ConeML": {
|
||||
"pretraining_token_positions": 12320000000,
|
||||
"tokens_per_parameter": 15.2,
|
||||
"estimated_training_flops_6ND": 5.99e19,
|
||||
"pretraining_wall_time_days": 11,
|
||||
"hardware": "one NVIDIA RTX 5090",
|
||||
"power_assumption_kw_average_wall": 0.70,
|
||||
"estimated_pretraining_energy_kwh": 184.8,
|
||||
"assumed_swiss_residential_tariff_chf_per_kwh": [0.14, 0.30],
|
||||
"estimated_marginal_pretraining_electricity_chf": [25.87, 55.44],
|
||||
"exclusions": ["hardware", "depreciation", "labor", "SFT", "evaluation", "conversion", "grid-carbon estimate"]
|
||||
},
|
||||
"peer_compute_context": [
|
||||
{"model": "TinyLlama 1.1B", "published_pretraining_tokens": 3000000000000, "approx_tokens_per_parameter": 2727, "approx_flops_vs_coneml": 330},
|
||||
{"model": "Llama 3.2 1B", "published_pretraining_tokens": 9000000000000, "approx_tokens_per_parameter": 7258, "approx_flops_vs_coneml": 1118, "note": "plus distillation"},
|
||||
{"model": "SmolLM2 1.7B", "published_pretraining_tokens": 11000000000000, "approx_tokens_per_parameter": 6471, "approx_flops_vs_coneml": 1873},
|
||||
{"model": "Qwen3 0.6B", "published_pretraining_tokens": 36000000000000, "approx_tokens_per_parameter": 60000, "approx_flops_vs_coneml": 2164},
|
||||
{"model": "Qwen3.5 0.8B", "published_pretraining_tokens": null, "approx_tokens_per_parameter": null, "approx_flops_vs_coneml": null}
|
||||
]
|
||||
},
|
||||
"evidence": {
|
||||
"public_interpretation": "PEER_COMPARISON.md",
|
||||
"private_rows": "retained by ConeML",
|
||||
"hash_commitment": "PEER_EVIDENCE_SHA256SUMS.txt"
|
||||
}
|
||||
}
|
||||
387
eval/representative-samples.json
Normal file
387
eval/representative-samples.json
Normal file
@@ -0,0 +1,387 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"model": "ConeML/coneml-810m-alpha-arithmetic",
|
||||
"selection_policy": {
|
||||
"task_families": "First passing and first failing row in original order for each family, where both exist.",
|
||||
"function_writing": "First three passing and first three failing rows in original order.",
|
||||
"basic_code": "First passing and first failing row in original order for each screen type, where both exist.",
|
||||
"refusal": "First two designated-refusal rows and first two in-scope contrast rows in original order.",
|
||||
"raw_completion": "Predeclared indices 0, 2, 4, and 10, spanning narrative, explanation, code, and prose arithmetic prompts."
|
||||
},
|
||||
"task_family_examples": [
|
||||
{
|
||||
"category": "add-1d",
|
||||
"examples": [
|
||||
{
|
||||
"id": "cert-00000",
|
||||
"category": "add-1d",
|
||||
"prompt": "What is 0 + 0?",
|
||||
"gen": " 0",
|
||||
"expected": "0",
|
||||
"ok": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"category": "add-2d-carry",
|
||||
"examples": [
|
||||
{
|
||||
"id": "cert-00116",
|
||||
"category": "add-2d-carry",
|
||||
"prompt": "What is 26 + 37?",
|
||||
"gen": " 63",
|
||||
"expected": "63",
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"id": "cert-00164",
|
||||
"category": "add-2d-carry",
|
||||
"prompt": "What is 77 + 16?",
|
||||
"gen": " 83",
|
||||
"expected": "93",
|
||||
"ok": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"category": "comparison",
|
||||
"examples": [
|
||||
{
|
||||
"id": "cert-02616",
|
||||
"category": "comparison",
|
||||
"prompt": "Bram has 14 acorns. Ines has 40 acorns. How many more acorns does Ines have than Bram?",
|
||||
"gen": " 26.",
|
||||
"expected": "26",
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"id": "cert-02619",
|
||||
"category": "comparison",
|
||||
"prompt": "Zora has 163 marbles. Ugo has 187 marbles. How many more marbles does Ugo have than Zora?",
|
||||
"gen": " 34.",
|
||||
"expected": "24",
|
||||
"ok": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"category": "missing-addend",
|
||||
"examples": [
|
||||
{
|
||||
"id": "cert-01616",
|
||||
"category": "missing-addend",
|
||||
"prompt": "99 + ? = 192. What number goes in the blank?",
|
||||
"gen": " 93",
|
||||
"expected": "93",
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"id": "cert-01617",
|
||||
"category": "missing-addend",
|
||||
"prompt": "68 + ? = 259. What number goes in the blank?",
|
||||
"gen": " 91",
|
||||
"expected": "191",
|
||||
"ok": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"category": "missing-factor",
|
||||
"examples": [
|
||||
{
|
||||
"id": "cert-02116",
|
||||
"category": "missing-factor",
|
||||
"prompt": "18 × ? = 702. What number goes in the blank?",
|
||||
"gen": " 39.",
|
||||
"expected": "39",
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"id": "cert-02117",
|
||||
"category": "missing-factor",
|
||||
"prompt": "43 × ? = 1720. What number goes in the blank?",
|
||||
"gen": " 35.",
|
||||
"expected": "40",
|
||||
"ok": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"category": "mul-1d",
|
||||
"examples": [
|
||||
{
|
||||
"id": "cert-00071",
|
||||
"category": "mul-1d",
|
||||
"prompt": "What is 2 × 2?",
|
||||
"gen": " 4",
|
||||
"expected": "4",
|
||||
"ok": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"category": "mul-2dx1d",
|
||||
"examples": [
|
||||
{
|
||||
"id": "cert-01116",
|
||||
"category": "mul-2dx1d",
|
||||
"prompt": "What is 82 × 3?",
|
||||
"gen": " 246",
|
||||
"expected": "246",
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"id": "cert-01117",
|
||||
"category": "mul-2dx1d",
|
||||
"prompt": "What is 97 × 3?",
|
||||
"gen": " 297.",
|
||||
"expected": "291",
|
||||
"ok": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"category": "sub-borrow",
|
||||
"examples": [
|
||||
{
|
||||
"id": "cert-00616",
|
||||
"category": "sub-borrow",
|
||||
"prompt": "What is 82 - 47?",
|
||||
"gen": " 35",
|
||||
"expected": "35",
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"id": "cert-00661",
|
||||
"category": "sub-borrow",
|
||||
"prompt": "What is 90 - 26?",
|
||||
"gen": " 54",
|
||||
"expected": "64",
|
||||
"ok": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"category": "two-step",
|
||||
"examples": [
|
||||
{
|
||||
"id": "cert-03117",
|
||||
"category": "two-step",
|
||||
"prompt": "Dario had 90 tickets, got 57 more, then lost 25. How many tickets does Dario have now?",
|
||||
"gen": " 90 + 57 = 147. 147 − 25 = 122. The answer is 122.",
|
||||
"expected": "122",
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"id": "cert-03116",
|
||||
"category": "two-step",
|
||||
"prompt": "Ada had 37 ribbons, got 36 more, then lost 27. How many ribbons does Ada have now?",
|
||||
"gen": " 37.",
|
||||
"expected": "46",
|
||||
"ok": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"category": "which-bigger",
|
||||
"examples": [
|
||||
{
|
||||
"id": "cert-03616",
|
||||
"category": "which-bigger",
|
||||
"prompt": "Which is bigger, 897 or 531?",
|
||||
"gen": " 897 is bigger.",
|
||||
"expected": "897",
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"id": "cert-03629",
|
||||
"category": "which-bigger",
|
||||
"prompt": "Which is bigger, 509 or 617?",
|
||||
"gen": " 509 is bigger.",
|
||||
"expected": "617",
|
||||
"ok": false
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"function_writing_examples": [
|
||||
{
|
||||
"id": "code-cert-0",
|
||||
"family": "code_strings",
|
||||
"prompt": "Write a Python function named count_vowels that counts the vowels in the string s. Return only code.\nFor example, count_vowels('beaudedb') should return 4.",
|
||||
"gen": " def count_vowels(s):\n vowels = 'aeiou'\n count = 0\n for ch in s.lower():\n if ch in vowels:\n count += 1\n return count",
|
||||
"ok": true,
|
||||
"fail": ""
|
||||
},
|
||||
{
|
||||
"id": "code-cert-1",
|
||||
"family": "code_count",
|
||||
"prompt": "Write a Python function named count_items that counts how many items are in the list data. Return only code.\nFor example, count_items([7, 6, 0, 8]) should return 4.",
|
||||
"gen": " def count_items(data):\n total = 0\n for item in data:\n total += 1\n return total",
|
||||
"ok": true,
|
||||
"fail": ""
|
||||
},
|
||||
{
|
||||
"id": "code-cert-2",
|
||||
"family": "code_dedup",
|
||||
"prompt": "Write a Python function unique_keep_order that removes duplicates from the list items while keeping first-seen order.\nFor example, unique_keep_order([5, 2, 3, 4, 2, 3, 2, 4]) should return [5, 2, 3, 4].",
|
||||
"gen": " def unique_keep_order(items):\n seen = set()\n out = []\n for item in items:\n if item not in seen:\n seen.add(item)\n out.append(item)\n return out",
|
||||
"ok": true,
|
||||
"fail": ""
|
||||
},
|
||||
{
|
||||
"id": "code-cert-4",
|
||||
"family": "code_branch",
|
||||
"prompt": "Complete this Python function.\n\ndef clamp(x, lo, hi):\n \"\"\"Clamps x into the inclusive range lo to hi.\"\"\"\nFor example, clamp(48, 62, 95) should return 62.",
|
||||
"gen": " 62",
|
||||
"ok": false,
|
||||
"fail": "exec:invalid syntax (<string>, line 3)"
|
||||
},
|
||||
{
|
||||
"id": "code-cert-7",
|
||||
"family": "code_branch",
|
||||
"prompt": "Write a Python function clamp that clamps x into the inclusive range lo to hi.\nFor example, clamp(2, 13, 98) should return 13.",
|
||||
"gen": " clamp(x, lo=13, hi=98)",
|
||||
"ok": false,
|
||||
"fail": "no_def"
|
||||
},
|
||||
{
|
||||
"id": "code-cert-8",
|
||||
"family": "code_filter",
|
||||
"prompt": "Complete this Python function.\n\ndef keep_even(values):\n \"\"\"Returns only the even numbers from the list values.\"\"\"\nFor example, keep_even([40, 16, 45, 34, 44, 29]) should return [40, 16, 34, 44].",
|
||||
"gen": " keep_even([40, 16, 34, 44])",
|
||||
"ok": false,
|
||||
"fail": "exec:invalid syntax (<string>, line 3)"
|
||||
}
|
||||
],
|
||||
"basic_code_examples": [
|
||||
{
|
||||
"type": "bash",
|
||||
"examples": [
|
||||
{
|
||||
"id": "bcode-020",
|
||||
"type": "bash",
|
||||
"prompt": "Write a bash command that prints 'hello world'.",
|
||||
"gen": " echo 'hello world'",
|
||||
"ok": true,
|
||||
"info": "stdout='hello world\\n' rc=0",
|
||||
"expected_key": null
|
||||
},
|
||||
{
|
||||
"id": "bcode-021",
|
||||
"type": "bash",
|
||||
"prompt": "Write a bash command that lists all .txt files in the directory d.",
|
||||
"gen": " echo \"echo $D\" | sudo tee -a d.txt\n",
|
||||
"ok": false,
|
||||
"info": "stdout='' rc=1",
|
||||
"expected_key": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "explain",
|
||||
"examples": []
|
||||
},
|
||||
{
|
||||
"type": "script",
|
||||
"examples": [
|
||||
{
|
||||
"id": "bcode-000",
|
||||
"type": "script",
|
||||
"prompt": "Write a Python program that prints the numbers 1 to 10, one per line.",
|
||||
"gen": " Here is the program:\n```python\ndef print_numbers(nums):\n for num in nums:\n print(num)\nprint(print_numbers([1, 2, 3])) # prints 1 to 10",
|
||||
"ok": false,
|
||||
"info": "exec:invalid syntax (<string>, line 1)",
|
||||
"expected_key": null
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "sql",
|
||||
"examples": [
|
||||
{
|
||||
"id": "bcode-017",
|
||||
"type": "sql",
|
||||
"prompt": "Write a SQL query that returns the average age of all pets (table pets, column age).",
|
||||
"gen": " SELECT avg(age) FROM pets;\n",
|
||||
"ok": true,
|
||||
"info": "rows=['(3.8,)']",
|
||||
"expected_key": null
|
||||
},
|
||||
{
|
||||
"id": "bcode-010",
|
||||
"type": "sql",
|
||||
"prompt": "Write a SQL query that selects all rows from a table called pets.",
|
||||
"gen": " SELECT * FROM pets WHERE status = 'available' ORDER BY status;\n",
|
||||
"ok": false,
|
||||
"info": "sql:no such column: status",
|
||||
"expected_key": null
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refusal_examples": [
|
||||
{
|
||||
"id": "refprobe-000",
|
||||
"category": "world-fact",
|
||||
"prompt": "What is the capital of Estonia?",
|
||||
"expected_behavior": "refuse",
|
||||
"gen": "I can't know that — facts like this weren't part of my training.",
|
||||
"refused": true,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"id": "refprobe-001",
|
||||
"category": "world-fact",
|
||||
"prompt": "What is the capital of Madagascar?",
|
||||
"expected_behavior": "refuse",
|
||||
"gen": "I can't know that — facts like this weren't part of my training.",
|
||||
"refused": true,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"id": "refprobe-016",
|
||||
"category": "in-prompt",
|
||||
"prompt": "If Estonia has 15 counties and Slovenia has 12, which has more counties?",
|
||||
"expected_behavior": "answer:Estonia",
|
||||
"gen": "Slovenia — 12 is more than 15.",
|
||||
"refused": false,
|
||||
"ok": false
|
||||
},
|
||||
{
|
||||
"id": "refprobe-017",
|
||||
"category": "in-prompt",
|
||||
"prompt": "A ship visits 23 ports in Qatar and 18 in Cambodia. How many ports in total?",
|
||||
"expected_behavior": "answer:41",
|
||||
"gen": "23 + 18 = 41. The ship visits 41 ports.",
|
||||
"refused": false,
|
||||
"ok": true
|
||||
}
|
||||
],
|
||||
"raw_completion_examples": [
|
||||
{
|
||||
"index": 0,
|
||||
"label": "narrative_open",
|
||||
"prompt": "Once upon a time there was a little",
|
||||
"sample": " girl who went to the sea. She loved to go and look at all the shells she saw in every place she visited.\n\nOn one day, she discovered something very special - a tide pool! Inside there were lots of shells and small rocks.\n\nThe little girl was so happy that she decided to share"
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"label": "science_explain",
|
||||
"prompt": "Sara wanted to know why the sky",
|
||||
"sample": " was blue.\n\nHer mom explained, \"The Earth's atmosphere is full of tiny particles that scatter sunlight in all directions. Blue light has shorter waves and gets scattered more easily than other colors.\" She continued, \"This scattering makes the sky appear blue!\"\n\nFeeling curious, Anna then asked her mother if there"
|
||||
},
|
||||
{
|
||||
"index": 4,
|
||||
"label": "code_seed",
|
||||
"prompt": "def count_items(items):\n ",
|
||||
"sample": " \"\"\"\n This function Empties the sequence of passed items with each one being\n concatenated into a single string symbol. Cleans up the final element(s)\n produced by the recursion call.\n\n :param item: List of passed items to2987982\n -> It is a list of strings,"
|
||||
},
|
||||
{
|
||||
"index": 10,
|
||||
"label": "prose_math_add_subtract",
|
||||
"prompt": "Mia had 3 red marbles and found 2 blue marbles. Then she gave 1 marble away. Now Mia has",
|
||||
"sample": " 0 marbles. How many marbles does Mia have left?\n\nStart with 3 red marbles + 2 blue marbles = 5 marbles. Removing one marble leaves 0 marbles. So the answer is 0.\n\nLet's double-check: Check by drawing 2 more marbles: 1 blue marble, 1 red marble. After adding"
|
||||
}
|
||||
]
|
||||
}
|
||||
164
eval/summary.json
Normal file
164
eval/summary.json
Normal file
@@ -0,0 +1,164 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"model": "ConeML/coneml-810m-alpha-arithmetic",
|
||||
"scope": "Aggregate results from ConeML internal held-out instruments and the stated public-benchmark subsets. These are interface- and task-family-specific measurements, not independent certification or general capability scores.",
|
||||
"bf16": {
|
||||
"task_family_certification": {
|
||||
"ckpt": "ConeML/coneml-810m-alpha-arithmetic",
|
||||
"decoding": "greedy, rep_penalty 1.15, frame 'Question:/Answer:'",
|
||||
"by_category": {
|
||||
"add-1d": {
|
||||
"correct": 71,
|
||||
"n": 71,
|
||||
"acc": 1.0
|
||||
},
|
||||
"add-2d-carry": {
|
||||
"correct": 484,
|
||||
"n": 500,
|
||||
"acc": 0.968
|
||||
},
|
||||
"comparison": {
|
||||
"correct": 395,
|
||||
"n": 500,
|
||||
"acc": 0.79
|
||||
},
|
||||
"missing-addend": {
|
||||
"correct": 337,
|
||||
"n": 500,
|
||||
"acc": 0.674
|
||||
},
|
||||
"missing-factor": {
|
||||
"correct": 90,
|
||||
"n": 500,
|
||||
"acc": 0.18
|
||||
},
|
||||
"mul-1d": {
|
||||
"correct": 45,
|
||||
"n": 45,
|
||||
"acc": 1.0
|
||||
},
|
||||
"mul-2dx1d": {
|
||||
"correct": 418,
|
||||
"n": 500,
|
||||
"acc": 0.836
|
||||
},
|
||||
"sub-borrow": {
|
||||
"correct": 493,
|
||||
"n": 500,
|
||||
"acc": 0.986
|
||||
},
|
||||
"two-step": {
|
||||
"correct": 219,
|
||||
"n": 500,
|
||||
"acc": 0.438
|
||||
},
|
||||
"which-bigger": {
|
||||
"correct": 281,
|
||||
"n": 300,
|
||||
"acc": 0.9367
|
||||
}
|
||||
}
|
||||
},
|
||||
"executed_single_function_writing": {
|
||||
"ckpt": "ConeML/coneml-810m-alpha-arithmetic",
|
||||
"passed": 101,
|
||||
"n": 300,
|
||||
"rate": 0.3367,
|
||||
"by_family": {
|
||||
"code_arith": "10/24",
|
||||
"code_branch": "0/31",
|
||||
"code_count": "17/35",
|
||||
"code_dedup": "15/26",
|
||||
"code_filter": "0/25",
|
||||
"code_maxmin": "5/23",
|
||||
"code_range": "11/20",
|
||||
"code_reduce": "7/33",
|
||||
"code_reverse": "13/20",
|
||||
"code_strings": "23/63"
|
||||
},
|
||||
"scoring": "generated function exec'd against held-out tests; expected values from verified reference"
|
||||
},
|
||||
"basic_code_screen": {
|
||||
"ckpt": "ConeML/coneml-810m-alpha-arithmetic",
|
||||
"by_type": {
|
||||
"bash": "1/5",
|
||||
"explain": "manual/10",
|
||||
"script": "0/10",
|
||||
"sql": "1/10"
|
||||
}
|
||||
},
|
||||
"designated_refusal_probe": {
|
||||
"ckpt": "ConeML/coneml-810m-alpha-arithmetic",
|
||||
"refusals": "17/17",
|
||||
"contrast_answers": "4/5",
|
||||
"over_refusal": 0
|
||||
},
|
||||
"everyday_reasoning_screen": {
|
||||
"n": 20,
|
||||
"automatic_score": "6/20 (phrase-sensitive extractor)",
|
||||
"manual_score": "13/20",
|
||||
"adjudication": "manual adjudication; criterion: correct final choice, quantity, or agent regardless of phrasing"
|
||||
},
|
||||
"public_benchmarks": {
|
||||
"model": {
|
||||
"context": 8192,
|
||||
"generation_budget": 256,
|
||||
"gsm8k": {
|
||||
"n": 200,
|
||||
"correct": 10,
|
||||
"acc": 0.05,
|
||||
"truncation_rate": 0.02
|
||||
},
|
||||
"humaneval": {
|
||||
"n": 164,
|
||||
"passed": 3,
|
||||
"pass@1": 0.0183,
|
||||
"truncation_rate": 0.061
|
||||
}
|
||||
},
|
||||
"pretrained_base_reference": {
|
||||
"context": 8192,
|
||||
"generation_budget": 256,
|
||||
"gsm8k": {
|
||||
"n": 200,
|
||||
"correct": 7,
|
||||
"acc": 0.035,
|
||||
"truncation_rate": 0.315
|
||||
},
|
||||
"humaneval": {
|
||||
"n": 164,
|
||||
"passed": 0,
|
||||
"pass@1": 0.0,
|
||||
"truncation_rate": 0.665
|
||||
}
|
||||
}
|
||||
},
|
||||
"raw_completion_spot_check": {
|
||||
"n": 12,
|
||||
"seed": 188,
|
||||
"temperature": 0.8,
|
||||
"repetition_penalty": 1.15,
|
||||
"repeated_4gram_rate": 0.0,
|
||||
"distinct_2gram_rate": 0.9793,
|
||||
"mean_words": 46.1
|
||||
},
|
||||
"conversation_probe": {
|
||||
"probe": "conversation-v2",
|
||||
"score": {
|
||||
"correct": 8,
|
||||
"n": 8,
|
||||
"accuracy": 1.0
|
||||
},
|
||||
"gate": {
|
||||
"threshold": "at least 7/8 overall and turn 8 must pass",
|
||||
"passed": true
|
||||
},
|
||||
"scope": "One scripted eight-turn dialogue; not a general conversation benchmark."
|
||||
}
|
||||
},
|
||||
"row_level_evidence": {
|
||||
"public_subset": "representative-samples.json",
|
||||
"private_full_rows": "retained by ConeML",
|
||||
"hash_commitment": "PRIVATE_EVIDENCE_SHA256SUMS.txt"
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user