ConeML/coneml-810m-alpha-arithmetic
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: ConeML/coneml-810m-alpha-arithmetic
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-10-13 19:16:26 +08:00
commit 189e25fdad
23 changed files with 164330 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

37
.gitattributes vendored Normal file
View File

@@ -0,0 +1,37 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
coneml-810m-alpha-arithmetic-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
coneml-810m-alpha-arithmetic-f16.gguf filter=lfs diff=lfs merge=lfs -text

16
LICENSE.md Normal file
View File

@@ -0,0 +1,16 @@
# License
Copyright 2026 ConeML.
This release is licensed under the Creative Commons
Attribution-NonCommercial 4.0 International license (CC BY-NC 4.0).
License terms: https://creativecommons.org/licenses/by-nc/4.0/legalcode
You may share and adapt the released material subject to the attribution
and non-commercial conditions in that license. Commercial use is not
granted by this release. The material is provided without warranties or
conditions beyond those required by applicable law.
For commercial evaluation or licensing enquiries, contact ConeML through
https://huggingface.co/ConeML.

16
Modelfile Normal file
View File

@@ -0,0 +1,16 @@
FROM ./coneml-810m-alpha-arithmetic-Q8_0.gguf
TEMPLATE """{{- range .Messages }}
{{- if eq .Role "system" }}System:
{{ .Content }}
{{- else if eq .Role "user" }}User:
{{ .Content }}
{{- else if eq .Role "assistant" }}Assistant:
{{ .Content }}<|endoftext|>
{{- end }}
{{- end }}Assistant:
"""
PARAMETER num_ctx 8192
PARAMETER repeat_penalty 1.15
PARAMETER stop "<|endoftext|>"

269
README.md Normal file
View File

@@ -0,0 +1,269 @@
---
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: text-generation
language:
- en
---
# ConeML 810M Alpha-Arithmetic
ConeML 810M Alpha-Arithmetic is an 810,112,512-parameter, decoder-only
language model trained from scratch and instruction-tuned for a narrow
small-number arithmetic profile. It shares its pretrained base with
`coneml-810m-alpha` and differs in fine-tuning composition.
This variant records the stronger carry/borrow profile of the pair. The
companion `coneml-810m-alpha` has substantially stronger function-writing
and two-step results. Results are specific to the documented prompt
formats and task families, not claims of general mathematical, coding,
reasoning, or factual-knowledge performance.
- Architecture: LLaMA-style; 28 layers, d=1536, 12 heads (3 KV), tied
embeddings, 8192 context, 32,768-token BPE tokenizer
- Pretraining corpus: 27.87B curated tokens
- Selected pretrained-base exposure: approximately 12.32B token
positions, or 15.2 tokens per parameter
- Pretraining wall time: approximately 11 days on one NVIDIA RTX 5090;
fine-tuning and evaluation are excluded
- Release `model.safetensors` SHA-256:
`e1de27f5f91ac43130ce2a57a4bf9a036b84b2e5b589789b5981ecf79489e0d1`
## 1. Measured profile
Across four held-out core lane sets—1-digit addition, 1-digit
multiplication, carry addition, and borrow subtraction—the model scores
1,093/1,116 (97.9%). Two of those four held-out sets scored 100%.
This is a result on four explicitly named, task-family-matched sets. It is
not a claim of 97.9% general arithmetic accuracy. ConeML internal
certification is not independent third-party certification.
| held-out task family | result | n |
|---|---:|---:|
| 2–3-digit addition with carry | 96.8% | 500 |
| 2-digit subtraction with borrow | 98.6% | 500 |
| 1-digit addition, including 19 zero-operand items | 100% | 71 |
| 1-digit multiplication | 100% | 45 |
| "Which is bigger, X or Y?" | 93.7% | 300 |
| comparison word problems | 79.0% | 500 |
| missing addend | 67.4% | 500 |
| two-step word problems | 43.8% | 500 |
| 2-digit × 1-digit multiplication | 83.6% | 500 |
| missing factor | 18.0% | 500 |
| Python function writing (single function, executed) | 33.7% | 300 |
Behavioral screens:
- Designated-refusal probe: 17/17 refusal prompts produced refusals, and
4/5 in-scope contrast prompts were answered correctly. The miss was an
incorrect comparison answer, not an over-refusal. This is a small
targeted policy test, not general epistemic calibration.
- Conversation-v2: 8/8 turns passed in one scripted dialogue covering
cross-turn arithmetic, story revision, comparison explanation, and one
unknown-personal-fact refusal. This is a single dialogue, not a general
conversation benchmark.
- Everyday-reasoning screen: 13/20 by manual adjudication; the
phrase-sensitive automatic scorer recorded 6/20.
## 2. Peer comparison and efficiency
The pair was tested against five post-trained peers on the same reduced
diagnostic screen, through each model's native instruction interface,
with greedy decoding, repetition penalty 1.15, and matched short-answer
budgets.
| model | parameters | mixed arithmetic (n=585) | executed functions (n=100) | transitive names d1/d3/d5 (n=32 each) | designated refusals (n=17) |
|---|---:|---:|---:|---:|---:|
| ConeML 810M Alpha | 0.81B | 421 (72.0%) | 83 | 23/15/15 | **13** |
| ConeML 810M Alpha-Arithmetic | 0.81B | **442 (75.6%)** | 35 | 26/17/16 | 11 |
| Qwen3.5 | 0.8B | 154 (26.3%) | 93 | 23/22/10 | 0 |
| Qwen3 | 0.6B | 223 (38.1%) | **98** | 18/10/9 | 1 |
| Llama 3.2 Instruct | 1.24B | 343 (58.6%) | 79 | 16/7/5 | 1 |
| TinyLlama Chat | 1.1B | 113 (19.3%) | 51 | 8/18/22 | 0 |
| SmolLM2 Instruct | 1.7B | 380 (65.0%) | 96 | 16/6/5 | 1 |
This is not a general leaderboard: the task families match ConeML's
trained surfaces. On this screen, the ConeML pair led mixed arithmetic;
ConeML 810M Alpha exceeded Llama 3.2 and TinyLlama on function writing
but trailed Qwen3.5, Qwen3, and SmolLM2. All models recorded zero
over-refusals on five in-scope contrasts.
On the question-form name-chain depth screen, ConeML 810M
Alpha-Arithmetic recorded 26/32, 17/32, and 16/32 at depths 1, 3, and 5;
ConeML 810M Alpha recorded 23/32, 15/32, and 15/32. The arithmetic
variant exceeded Qwen3 and Llama 3.2 at all three shown depths and
Qwen3.5 at depths 1 and 5, while Qwen3.5 led it at depth 3 and TinyLlama
led the group at depth 5. Entity-chain controls were mixed and are
reported in the full table. Chance is 1/(depth+1); this is an
exact-selection surface test, not a claim of general reasoning.
Qwen3.5 with thinking enabled reached 451/585 (77.1%), compared with
this release's 442/585 (75.6%), while using at least 17.6 times the
generated-token budget per item and approximately 24 times the recorded
wall time. Runtime ratios are implementation-specific.
ConeML's selected base consumed 12.32B token positions—15.2 tokens per
parameter and approximately 5.99e19 training FLOPs under the `6ND`
convention. Pretraining took approximately 11 days on one RTX 5090.
Assuming, rather than claiming to have metered, a 0.70 kW average wall
draw gives 184.8 kWh; at CHF 0.14–0.30/kWh, that is approximately
CHF 26–55 of marginal pretraining electricity. This excludes hardware,
labor, SFT, evaluation, conversion, and emissions accounting.
The complete category table, peer wins, adverse base result, evaluated
revisions, assumptions, and frozen evidence commitments are in
[`eval/PEER_COMPARISON.md`](eval/PEER_COMPARISON.md).
## 3. Standard benchmark limits
Zero-shot, greedy decoding with a 256-token generation cap. GSM8K uses
strict final-number matching; HumanEval reports pass@1 from executed
tests.
| benchmark | this model | same base, no fine-tune |
|---|---:|---:|
| GSM8K (test, first 200) | 10/200 (5.0%) | 7/200 (3.5%) |
| HumanEval (pass@1, tests executed) | 3/164 (1.8%) | 0/164 (0.0%) |
These results are weak and define the boundary of the specialization:
the four-lane arithmetic profile does not transfer to broad GSM8K or
HumanEval performance.
## 4. Supported prompt formats
Evaluated decoding is greedy with repetition penalty 1.15, as pinned in
`generation_config.json`.
```text
# task format (single turn)
Question: <task>
Answer:
# dialogue format (chat_template.jinja)
User:
<message>
Assistant:
```
Dialogue generation ends at `<|endoftext|>`. Prompting outside these
formats, including paraphrases, was not systematically evaluated except
for the published raw-completion spot check.
## 5. Quantized variants
Q8_0 GGUF was evaluated separately under llama.cpp. Its runtime and stop
handling differ from the bf16 harness, so score differences cannot be
attributed solely to quantization.
| held-out task family | Q8_0 |
|---|---:|
| subtraction with borrow | 98.6% (n=500) |
| addition with carry | 92.6% (n=500) |
| 1-digit multiplication | 100% (n=45) |
| two-step word problems | 32.6% (n=500) |
For local GGUF inference:
```bash
llama-cli \
-m coneml-810m-alpha-arithmetic-Q8_0.gguf \
-p $'Question: What is 47 + 38?\nAnswer:' \
-n 32 --repeat-penalty 1.15 --no-conversation --single-turn
```
For Ollama, download `Modelfile` beside the Q8_0 file, then run:
```bash
ollama create coneml-810m-alpha-arithmetic -f Modelfile
ollama run coneml-810m-alpha-arithmetic "What is 47 + 38?"
```
## 6. Loading with Transformers
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "ConeML/coneml-810m-alpha-arithmetic"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
dtype=torch.bfloat16,
device_map="auto",
)
prompt = "Question: What is 47 + 38?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
**inputs,
max_new_tokens=32,
do_sample=False,
repetition_penalty=1.15,
)
answer = tokenizer.decode(
output[0][inputs["input_ids"].shape[-1]:],
skip_special_tokens=True,
)
print(answer)
```
This is a text-only causal language model. Load it with
`AutoModelForCausalLM` or `LlamaForCausalLM`, not a multimodal model class.
## 7. Intended use and safety
This release is intended for research and constrained local workflows
inside its measured small-number arithmetic envelope. It is not a general
mathematics system. Validate outputs whenever exactness matters.
It is not intended for factual retrieval, high-stakes medical, legal, or
financial decisions, autonomous action, or unsandboxed execution of
generated code. Generated code may be incomplete or unsafe; inspect and
test it in an isolated environment.
## 8. Known limitations
- The measured strength is narrow. Missing factor scored 18.0%, two-step
word problems 43.8%, 2-digit × 1-digit multiplication 83.6%, and
executed single-function writing 33.7%.
- Not a code model: complete programs scored 0/10, executable SQL 1/10,
and bash 1/5.
- The model was not evaluated as a factual-QA system and refused all six
real-world factual prompts in the designated-refusal probe.
- Money/decimal change-making scored 0/3 on the held-out screen; the model
also showed yes-bias on feasibility questions.
- Multi-step GSM-class reasoning remained weak.
- Frameless prompting was spot-checked, not systematically certified.
## 9. Evidence and reproducibility
The `eval/` directory publishes:
- `summary.json`: all aggregate bf16 results used above;
- `representative-samples.json`: deterministic, rule-selected examples
from ConeML-owned probes;
- `EVALUATION_METHODOLOGY.md`: evaluation definitions and scope;
- separate Q8_0 and focused aggregate results; and
- `PRIVATE_EVIDENCE_SHA256SUMS.txt`: commitments to the complete
row-level evidence retained by ConeML.
- `peer-comparison-summary.json`, `PEER_COMPARISON.md`, and
`PEER_EVIDENCE_SHA256SUMS.txt`: the peer results, interpretation, and
commitments to the retained peer-generation evidence.
Complete proprietary probe rows and generations are retained privately to
avoid releasing the full measurement instruments. The hashes permit later
disclosure to be checked against the evidence frozen at release time.
Public GSM8K and HumanEval rows are not redistributed; only aggregate
results are published. `SHA256SUMS.txt` covers every shipped file.
## 10. Research license and commercial work
Released for non-commercial use under CC BY-NC 4.0. Commercial use is not
granted by this release. See `LICENSE.md`.
For commercial evaluation, model-engineering, or licensing enquiries,
contact [contact@coneml.com](mailto:contact@coneml.com) or visit
[ConeML for Organizations](https://coneml.com/organizations/). Commercial availability does not
imply that this alpha checkpoint is suitable for an unvalidated
production workflow.

21
SHA256SUMS.txt Normal file
View File

@@ -0,0 +1,21 @@
846a8e2297580b3c8d57f69a07d88833d370f976010b31d9fc9b93e9a3f04464 LICENSE.md
1756d02d239f8e8342a1aa513248f9d35b9a5168c77d07619e64a90e20580434 Modelfile
3f7f2da2b02a6a2b72fb2244b563a1b74f271e6c0d4fc64527795c3fc3d60042 README.md
fdf72939738a346a2134cd5ede1e9921d52d4d85ac0449249118322de48447e7 chat_template.jinja
147f479696718b5665f3aea3a7ae312cc6850d998246141f6ecbbd7e395bce2d coneml-810m-alpha-arithmetic-Q8_0.gguf
5898b62c447961cb7eb60fc538fc9f4b0f0bdda840fda87a861d01c245085c31 coneml-810m-alpha-arithmetic-f16.gguf
6cd7853f16b53fdb6bcc44438be39c33726379dc9848798fdcc81ba7f7555a58 config.json
9637d2b8ef44c95226638a31b7dea6fb3b6f817a297a1a25098f8215b0435489 conversion.json
83b8f89b84387d8c2b22bc2b5d8c20a564fea9f79c7f6e6ab4fa9acee38040c5 eval/EVALUATION_METHODOLOGY.md
e0b43abe260b588cdf50181b5e396f21abb4b69da151bfa4961539be128cd08d eval/PEER_COMPARISON.md
0f134f0b207601ef2d5650bfe4fd56e810e1c6e32938bdde94035ed08df95165 eval/PEER_EVIDENCE_SHA256SUMS.txt
e5722a62e0d7bf0d4d1ced1626cd5bc49381380aeeea671f6f40a4705740e088 eval/PRIVATE_EVIDENCE_SHA256SUMS.txt
f57c01a959c7944ac2e5bd516985115f67409b81445aca8a857c131f7b1f459e eval/certification-Q8_0.json
4388d66b5feeae2147beebc02967d536ec06a7d284ca245b119fa0dcc1b8cf8c eval/focused-summary.json
ce18163c651681a1c4ed2ff6cdc8644494f00a8109a96cc3d0b8768bb9a4c6fc eval/peer-comparison-summary.json
8bfd98eac2f9309c7481115b91ae47b8f01c9e3b44c80b2bb233a5be6947e8cc eval/representative-samples.json
ed8c2a178bf3268b89f325b36538b71aef54251adb53396e259f9534e6264504 eval/summary.json
2e2ddba5318c74a4fcc31cfdfdc6e067564176b6404da484932f95a7fdc9712a generation_config.json
e1de27f5f91ac43130ce2a57a4bf9a036b84b2e5b589789b5981ecf79489e0d1 model.safetensors
dd9e18c94d2206bb6c3e26caccc98f5b72dd930784cc37e989439b79ceacbf47 tokenizer.json
88a2d6096fc775fca098fa3d8f4be827b498d21af168e24ba3bd0509b8a63c59 tokenizer_config.json

8
chat_template.jinja Normal file
View File

@@ -0,0 +1,8 @@
{% for message in messages %}{% if message['role'] == 'system' %}System:
{{ message['content'] }}
{% elif message['role'] == 'user' %}User:
{{ message['content'] }}
{% elif message['role'] == 'assistant' %}Assistant:
{{ message['content'] }}{% if not loop.last %}{{ eos_token }}
{% endif %}{% endif %}{% endfor %}{% if add_generation_prompt %}Assistant:
{% endif %}

3
coneml-810m-alpha-arithmetic-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:147f479696718b5665f3aea3a7ae312cc6850d998246141f6ecbbd7e395bce2d
size 862157472

3
coneml-810m-alpha-arithmetic-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5898b62c447961cb7eb60fc538fc9f4b0f0bdda840fda87a861d01c245085c31
size 1621555872

32
config.json Normal file
View File

@@ -0,0 +1,32 @@
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "bfloat16",
"eos_token_id": 0,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 1536,
"initializer_range": 0.02,
"intermediate_size": 4608,
"max_position_embeddings": 8192,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 12,
"num_hidden_layers": 28,
"num_key_value_heads": 3,
"pad_token_id": 0,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_parameters": {
"rope_theta": 1000000,
"rope_type": "default"
},
"tie_word_embeddings": true,
"transformers_version": "5.12.1",
"use_cache": true,
"vocab_size": 32768
}

7
conversion.json Normal file
View File

@@ -0,0 +1,7 @@
{
"artifact": "ConeML/coneml-810m-alpha-arithmetic",
"conversion": "weights-only safetensors export",
"dtype": "bfloat16",
"missing_keys": [],
"unexpected_keys": []
}

59
eval/EVALUATION_METHODOLOGY.md Normal file
View File

@@ -0,0 +1,59 @@
# ConeML 810M — evaluation methodology and variant lineage (public record)
Two released variants share one from-scratch 810M base checkpoint and
differ only in supervised fine-tuning; both were evaluated with the same
bf16 instruments.
- `coneml-810m-alpha` — function-writing / mixed-arithmetic profile
- `coneml-810m-alpha-arithmetic` — carry/borrow arithmetic variant
## Methodology
- **ConeML internal certification**: large-n (300–500 per task family)
batteries of unseen instances in the trained prompt formats, greedy
decoding, repetition
penalty 1.15. Code tasks are scored by EXECUTING the generated function
against held-out tests; SQL by executing against a live sqlite database;
scripts by captured stdout. No pattern-match scoring for anything
execution can decide.
- **Behavior probes**: held-out designated-refusal prompts plus in-scope
contrast pairs in matching wording; one separately rerun, scripted
eight-turn conversation; and an everyday-reasoning screen with manual
adjudication. Aggregate results and deterministic representative
ConeML-owned rows are public; complete rows are retained privately.
- **Public benchmarks**: GSM8K (first 200, zero-shot, strict final-number
match) and HumanEval (pass@1, tests executed), identical harness across
variants and the pretrained base. Aggregate results are public;
benchmark rows are not redistributed.
- **Quantized variants are evaluated separately**: each shipped GGUF carries
its own result table. The GGUF and bf16 harnesses differ in runtime and
stop handling, so score gaps are not attributed solely to quantization.
## Why two variants
Under fixed architecture and base checkpoint, modest differences in
fine-tuning composition produced large, cross-surface capability
tradeoffs (e.g., one build scores 96.8% on held-out carry addition while
the other scores 79.0% on held-out two-step word problems and 69.0% on
held-out function writing). Rather than presenting one checkpoint as
dominant, each variant ships with its measured envelope and a
recommendation for which profile
fits which pipeline. A checkpoint's profile is measured, never inferred.
## Shared limitations
Function-writing is evaluated for single Python functions only — neither
variant is a general code model (complete programs, executable SQL, and
bash scored near zero for both and are stated out of scope). Both variants
refused every designated refusal prompt, including real factual questions;
this targeted behavior is not evidence of general epistemic calibration.
Multi-step GSM-class reasoning is weak in every variant tested this cycle.
## Evidence disclosure
The public folders contain aggregate result tables, deterministic
representative examples from ConeML-owned instruments, and SHA-256
commitments to the complete row-level evidence. Full proprietary probe
rows are retained by ConeML rather than distributed. This preserves the
measurement instruments while allowing later disclosure to be checked
against the evidence frozen for this release.

140
eval/PEER_COMPARISON.md Normal file
View File

@@ -0,0 +1,140 @@
# ConeML 810M peer diagnostic and resource context
Date: 2026-07-30
This is an interface-specific diagnostic screen, not a neutral ranking of
general model capability. Each instruction-tuned model was evaluated
through its own instruction interface. All models used greedy decoding
with repetition penalty 1.15. Arithmetic and function-writing used
matched short-answer generation budgets.
The task families match ConeML's supervised training surfaces. That gives
the comparison practical value for locating the release's envelope, but
it also means these results must not be generalized to unrelated tasks.
Standard GSM8K and HumanEval results remain separately disclosed in the
model cards.
## Instruct-model results
| measured surface | ConeML 810M Alpha | ConeML 810M Alpha-Arithmetic | Qwen3.5 0.8B | Qwen3 0.6B | Llama 3.2 1B Instruct | TinyLlama Chat | SmolLM2 1.7B Instruct |
|---|---:|---:|---:|---:|---:|---:|---:|
| mixed arithmetic screen (n=585) | 421 (72.0%) | **442 (75.6%)** | 154 (26.3%) | 223 (38.1%) | 343 (58.6%) | 113 (19.3%) | 380 (65.0%) |
| four core arithmetic lanes (n=225) | 177 (78.7%) | **221 (98.2%)** | 150 (66.7%) | 208 (92.4%) | **221 (98.2%)** | 92 (40.9%) | **221 (98.2%)** |
| executed single functions (n=100) | 83 | 35 | 93 | **98** | 79 | 51 | 96 |
| transitive names, depth 1/3/5 (n=32 each) | 23/15/15 | 26/17/16 | 23/22/10 | 18/10/9 | 16/7/5 | 8/18/22 | 16/6/5 |
| transitive entities, depth 1/3/5 (n=32 each) | 15/11/10 | 16/11/12 | 18/16/14 | 14/17/11 | 0/6/7 | 14/15/11 | 22/21/10 |
| designated-refusal prompts (n=17) | **13** | 11 | 0 | 1 | 1 | 0 | 1 |
| over-refusals on contrasts (n=5) | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
The honest result is mixed:
- At matched short-answer budgets, the ConeML pair led this post-trained
peer group on the broader arithmetic screen.
- On the four core arithmetic lanes, the Arithmetic variant tied Llama
3.2 Instruct and SmolLM2 at this reduced sample size. Its separate
full-size internal result is 1,093/1,116 (97.9%).
- ConeML 810M Alpha exceeded Llama 3.2 Instruct and TinyLlama Chat on the
executed function screen, but Qwen3.5, Qwen3, and SmolLM2 scored higher.
- On name-chain transitive selection, ConeML 810M Alpha-Arithmetic
recorded 26/32, 17/32,
and 16/32 at depths 1, 3, and 5. It exceeded Qwen3 and Llama 3.2 at all
three shown depths and exceeded Qwen3.5 at depths 1 and 5; Qwen3.5 led
it at depth 3, and TinyLlama led the group at depth 5.
- Entity-chain transfer was mixed: ConeML 810M Alpha recorded 15/11/10
and ConeML 810M Alpha-Arithmetic 16/11/12 at depths 1/3/5, while different peers led each
depth. Chance is 1/(depth+1). The non-monotonic peer rows reinforce
that this is an exact-selection surface test, not proof of general
reasoning depth.
- The refusal row measures a trained response policy on designated
prompts, not factual correctness or general epistemic calibration.
The peer-harness refusal results above differ from the dedicated ConeML
probe's 17/17 for each release because the generation loops differ. Both
measurements are reported rather than merged.
## Qwen3.5 thinking-mode sensitivity
Qwen3.5 0.8B was also evaluated with thinking enabled on the same 585
arithmetic items:
| metric | Qwen3.5 thinking | Qwen3.5 short-answer | ConeML 810M Alpha-Arithmetic short-answer |
|---|---:|---:|---:|
| accuracy | 451/585 (77.1%) | 154/585 (26.3%) | 442/585 (75.6%) |
| generated tokens per item | mean 844; median 699 | at most 48 | at most 48 |
| p95 / generation cap | 1,536 / 1,536 | 48 / 48 | 48 / 48 |
| truncation rate | 138/585 (23.6%) | approximately 0% | approximately 0% |
| mean wall time per item | 2.43 s | approximately 0.1 s | approximately 0.1 s |
| correct answers per 1,000 generated tokens | 0.91 | 5.5 | approximately 15.7 or higher |
Thinking mode recovered Qwen3.5 to statistical parity with ConeML 810M
Alpha-Arithmetic on this screen, while using at least 17.6 times the
per-answer generation budget and approximately 24 times the measured wall
time. Wall-time ratios are specific to the recorded hardware, batching,
and implementation. This is an inference-cost comparison, not a claim
that thinking mode is intrinsically inferior.
## Against-interest base result
On the same 585 arithmetic items, through a task frame native to neither
base model, Qwen3.5 0.8B Base scored 492/585 (84.1%), while ConeML base
checkpoint 188 scored 156/585 (26.7%). This result is included because it
prevents an absolute-superiority reading: high-exposure base models may
already contain strong task-formatted behavior, and post-training can
move capability between output surfaces.
## Training-resource context
ConeML's selected base consumed approximately 12.32B token positions:
15.2 tokens per parameter and approximately 5.99e19 training FLOPs under
the `6 × parameters × tokens` convention.
| model family | disclosed pretraining tokens | approximate tokens/parameter | approximate training FLOPs vs ConeML |
|---|---:|---:|---:|
| ConeML 810M | 12.32B | 15.2 | 1× |
| TinyLlama 1.1B | 3T | 2,727 | 330× |
| Llama 3.2 1B | up to 9T | 7,258 | 1,118×, plus distillation |
| SmolLM2 1.7B | 11T | 6,471 | 1,873× |
| Qwen3 0.6B | 36T | 60,000 | 2,164× |
| Qwen3.5 0.8B | not disclosed | not stated | not stated |
These are estimated pretraining FLOP ratios, not historical electricity
or monetary costs for the peer models.
ConeML pretraining took approximately 11 days on one local RTX 5090. At
an explicitly assumed average wall draw of 0.70 kW, that corresponds to
184.8 kWh. Applying an assumed Swiss residential tariff range of
CHF 0.14–0.30/kWh gives approximately CHF 26–55 of marginal pretraining
electricity.
| ConeML pretraining unit | estimate |
|---|---:|
| average throughput | 12,963 token positions/s |
| wall time per billion token positions | 21.4 h |
| energy per billion token positions | 15.0 kWh |
| energy per million token positions | 15 Wh |
| estimated wall energy per token position | 0.054 J |
| marginal electricity per billion token positions | CHF 2.10–4.50 |
The energy and cost figures are estimates derived from the stated power
and tariff assumptions, not meter readings. They exclude hardware,
depreciation, labor, supervised fine-tuning, evaluation, conversion,
datacenter PUE, and carbon intensity. No emissions claim is made.
## Evaluated revisions
- `Qwen/Qwen3.5-0.8B@2fc06364715b967f1860aea9cf38778875588b17`
- `Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca`
- `unsloth/Llama-3.2-1B-Instruct@5a8abab4a5d6f164389b1079fb721cfab8d7126c`
- `TinyLlama/TinyLlama-1.1B-Chat-v1.0@fe8a4ea1ffedaf415f4da2f062534de366a451e6`
- `HuggingFaceTB/SmolLM2-1.7B-Instruct@31b70e2e869a7173562077fd711b654946d38674`
The exact aggregates are in `peer-comparison-summary.json`. Complete
generation rows are retained privately; their frozen SHA-256 commitments
are published in `PEER_EVIDENCE_SHA256SUMS.txt`.
## Primary resource disclosures
- Qwen3 pretraining: [Qwen3 release post](https://qwenlm.github.io/blog/qwen3/)
- Llama 3.2 token count and distillation: [Meta Llama 3.2 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md)
- TinyLlama token count: [TinyLlama model card](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)
- SmolLM2 token count: [Hugging Face SmolLM2 model card](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B)

35
eval/PEER_EVIDENCE_SHA256SUMS.txt Normal file
View File

@@ -0,0 +1,35 @@
96a805aed179f77d9f1959216f9a39e8bf8d50dea5465eeea1eb24e96502a6af SOURCE_PEER_COMPARISON_V2.md
a5d6cd4d7a2643b316cefb1dcb6f43296bb69da4064dbc3b3ab4a123bf36b2b1 base-reference/coneml-base188.json
83643ce110655e708d50f4eb3ee29ab08ab8e003768eb5b5da9a3e545bc9b8ef base-reference/coneml-base188.rows.jsonl
ca387426c55ac72d50041e2f082bd4cdc747eedb51deca7929cf7a4acea2c5b6 base-reference/coneml-native-transitive.json
43ef584d37c6a276b190aeb741597b54bfebbf7bbd9c37df5d82a2e15a8ee2d1 base-reference/qwen35-0.8b-base.json
25de33180d4073777d29871882b57f1556733e9a8c1e10eba14a8cac062b5c35 base-reference/qwen35-0.8b-base.rows.jsonl
81b9effe8a6f808b5bf9dbf639ec88235004c6c270b5275e382316adb07ff9d1 instruct/coneml-810m-alpha-transchat.json
4b2c433ecf35c6dd409b9d09e2cb858e1d68e179d189dd9074e8ed681a9be44a instruct/coneml-810m-alpha-transchat.rows.jsonl
2003c73a54602f3a30980d6c7f85feef39b63fd11cef9fc20a90044d3569ba77 instruct/coneml-810m-alpha.json
8da107545bd6717c7795ff4e827698e5e269d451f2390f24dccc516fc390ab44 instruct/coneml-810m-alpha.rows.jsonl
aea504626bbd9f7d8daf576777fe82fa4c57947a30cd654953889edf8247101f instruct/coneml-810m-arithmetic-transchat.json
5b8864c9810d556a8bc2b09a43a2b34f69c0fad6889c8c04b63717c92b6f3c27 instruct/coneml-810m-arithmetic-transchat.rows.jsonl
77d514c14e14557c4ac30e8a8c3f4c5887f0db0dff39b8ea14baea164327d027 instruct/coneml-810m-arithmetic.json
a60ed3e533c0c4a77b34a21f80b8df8de85e3df074f3cacdc3bbd4b5e0741cce instruct/coneml-810m-arithmetic.rows.jsonl
aa35b93f46778ffb22351ac411f2fc04e645775e416b5a9423f925bb6d6b1609 instruct/llama32-1b-instruct-transchat.json
ee514c35d86f27172a004325b26da915f869164adae59edf8d95b01d958e683f instruct/llama32-1b-instruct-transchat.rows.jsonl
f4553ef34844e55364a87f631fac03a3545acbb14cd12dcf182f16151b5e5f8a instruct/llama32-1b-instruct.json
d5c0fd8b00b50754ac7dede96081afeaf3143795e005134cf5543483859340bf instruct/llama32-1b-instruct.rows.jsonl
a61b56681810410dfc3a8d8e758b8ccd11d01ebcce2d963d6bdbca4644c2cee9 instruct/qwen3-0.6b-transchat.json
f7c763441edc0d5e37a34902a3cc4efcacc6a35b47056adb9356879a4407cb31 instruct/qwen3-0.6b-transchat.rows.jsonl
3f34342d79095a5754512ebf57192fdf98a16b364a250712993ac1da0b65e252 instruct/qwen3-0.6b.json
05847f4187651780008331a85c9cd5fa84d3b636320c9a809fb3c307f1a230d6 instruct/qwen3-0.6b.rows.jsonl
21984e00a4a6da3d2887138ede25c03e9007fee0be41de462ef58dc58db89554 instruct/qwen35-0.8b-transchat.json
48a53b6bf3829c207953b1075160df9f6702b1fc2162426577aa69530856e555 instruct/qwen35-0.8b-transchat.rows.jsonl
fa11fbf67c047ac541ecd38571867f9fe68bfb8aec12b84f2addda8a6d561642 instruct/qwen35-0.8b.json
74a6debf6ae55db9895f556e654b6905fde871358a94c9c4a2730a418b0e2608 instruct/qwen35-0.8b.rows.jsonl
72598668bfce72497ac67a4d83d1cad7538c63ec933e8a1d1198876d85dc8a67 instruct/qwen35-thinking-sensitivity.json
8917677a4c9d29124c5ade229f7080bbd0ccb83471c3d7d39a6f958224596a8f instruct/smollm2-1.7b-instruct-transchat.json
deb9eb805475e4ad2d158283b469830138fdf42ee0c9a903bf3611b43a7869ff instruct/smollm2-1.7b-instruct-transchat.rows.jsonl
94a3440cdaae14eb27beaf36a1a0103dab4a6d292bc382981f0514a82a93ed0f instruct/smollm2-1.7b-instruct.json
1b06370ad7b940ed05b59ffe4acd19035341cfe078ed7b01d59d168abc8d0c71 instruct/smollm2-1.7b-instruct.rows.jsonl
bdd7cf2867643439e90a3e04cb209e6130839af9422e492bcd2eb1c2cba4556e instruct/tinyllama-chat-transchat.json
472aad0c169c4f943fbc9a8367d65fa2d5c41acbbee66d099e16a286f8676d1c instruct/tinyllama-chat-transchat.rows.jsonl
3a7c2e7ebcaa78b4b7d9b6d84a5b90a495f260f97287bad3282d54fc84e0c009 instruct/tinyllama-chat.json
5926a31e926279f8d9459d19cbbe9730a06071bb8b4029c2f9174804ab447943 instruct/tinyllama-chat.rows.jsonl

10
eval/PRIVATE_EVIDENCE_SHA256SUMS.txt Normal file
View File

@@ -0,0 +1,10 @@
532b8cd9c9f7eb7cd047aac26dfe6a99880503a7b09bf79314287c28cb908ce4 base188-public-benchmarks.json
9ae0002c88cd5972d21fe119734421f03f8ffcaa412002476ae5b0244bc302b7 basic-code.json
83ff212bfd15d342856cf1ececf46648d458b1a0010afc1538b3a87b7ef1b86b certification-v1.json
bed82d5fefc647f91d53cd22376fbb3b598ea417e317e698629bc82d07b6479c code-cert.json
3b263314c8690fb912a03fc87b5d81dd38e499a6f3ef4353dee124239572a826 conversation.log
ebca66bd78a552bf8e70498f5b38a778e27894afb2f47c122da29195d85dba3e conversation-v2.json
e7c6a5a5e7b7cb8c35cba5545cc56e79e01571e550a5958cad0ad3e9a0d1b9d6 everyday-manual-adjudication.json
3b698faa42fe5b556abcb7ba4ac39866a94ffd2413dcd5987251e9c5b46b1b91 public-benchmarks.json
3e28a81a3218e97503a4f572c5dc8cedd4943b02bf3de046614661f0d88bb247 raw-probe.jsonl
66f34c9e24e01a3e26db9faaf3606e510eb42639cd32169e500e17d1a2d53b38 refusal-probe.json

55
eval/certification-Q8_0.json Normal file
View File

@@ -0,0 +1,55 @@
{
"by_category": {
"add-1d": {
"correct": 70,
"n": 71,
"acc": 0.9859
},
"add-2d-carry": {
"correct": 463,
"n": 500,
"acc": 0.926
},
"comparison": {
"correct": 363,
"n": 500,
"acc": 0.726
},
"missing-addend": {
"correct": 337,
"n": 500,
"acc": 0.674
},
"missing-factor": {
"correct": 92,
"n": 500,
"acc": 0.184
},
"mul-1d": {
"correct": 45,
"n": 45,
"acc": 1.0
},
"mul-2dx1d": {
"correct": 398,
"n": 500,
"acc": 0.796
},
"sub-borrow": {
"correct": 493,
"n": 500,
"acc": 0.986
},
"two-step": {
"correct": 163,
"n": 500,
"acc": 0.326
},
"which-bigger": {
"correct": 277,
"n": 300,
"acc": 0.9233
}
},
"runtime": "llama.cpp Q8_0 aggregate summary"
}

25
eval/focused-summary.json Normal file
View File

@@ -0,0 +1,25 @@
{
"ckpt": "ConeML/coneml-810m-alpha-arithmetic",
"math_200": 138,
"everyday_20_auto": 6,
"by_category": {
"everyday::causal-evidence": "0/3",
"everyday::everyday-arithmetic": "2/2",
"everyday::money-change": "0/3",
"everyday::object-choice": "2/3",
"everyday::practical-judgment": "1/2",
"everyday::quantity-judgment": "0/2",
"everyday::temporal-order": "0/2",
"everyday::time-margin": "1/3",
"math::add-1d": "20/20",
"math::add-2d-carry": "30/30",
"math::missing-addend": "18/20",
"math::mul": "20/20",
"math::sub-2d-borrow": "29/30",
"math::transitive": "0/20",
"math::word-2step": "2/40",
"math::word-compare": "19/20"
},
"math_200_corrected": 154,
"correction": "transitive items scored by expected-name match on first line; original scorer extracted numbers only and zeroed all transitive rows"
}

140
eval/peer-comparison-summary.json Normal file
View File

@@ -0,0 +1,140 @@
{
"schema_version": 1,
"date": "2026-07-30",
"scope": "Interface-specific diagnostic screen, not a neutral ranking of general model capability. Each instruct model used its native instruction interface. Arithmetic and function-writing used matched short-answer generation budgets.",
"decoding": {
"strategy": "greedy",
"repetition_penalty": 1.15,
"arithmetic_n": 585,
"core_arithmetic_n": 225,
"function_writing_n": 100,
"transitive_n_per_family_depth": 32,
"designated_refusal_n": 17,
"refusal_contrast_n": 5
},
"evaluated_revisions": {
"ConeML/coneml-810m-alpha": "local release artifact e5df8f2edd773ad079ba00d383ace81b96cbb9579d64c570db505f9efc032b76",
"ConeML/coneml-810m-alpha-arithmetic": "local release artifact e1de27f5f91ac43130ce2a57a4bf9a036b84b2e5b589789b5981ecf79489e0d1",
"Qwen/Qwen3.5-0.8B": "2fc06364715b967f1860aea9cf38778875588b17",
"Qwen/Qwen3-0.6B": "c1899de289a04d12100db370d81485cdf75e47ca",
"unsloth/Llama-3.2-1B-Instruct": "5a8abab4a5d6f164389b1079fb721cfab8d7126c",
"TinyLlama/TinyLlama-1.1B-Chat-v1.0": "fe8a4ea1ffedaf415f4da2f062534de366a451e6",
"HuggingFaceTB/SmolLM2-1.7B-Instruct": "31b70e2e869a7173562077fd711b654946d38674"
},
"instruct_results": [
{
"model": "ConeML/coneml-810m-alpha",
"parameters_billions": 0.8101,
"arithmetic_mixed": {"correct": 421, "n": 585, "accuracy": 0.7197},
"arithmetic_core_four_lanes": {"correct": 177, "n": 225, "accuracy": 0.7867},
"executed_functions": {"correct": 83, "n": 100, "accuracy": 0.83},
"transitive_question_form": {"names_d1_d3_d5": [23, 15, 15], "entities_d1_d3_d5": [15, 11, 10], "n_each": 32},
"designated_refusals": {"correct": 13, "n": 17, "accuracy": 0.7647},
"over_refusals": {"count": 0, "n": 5}
},
{
"model": "ConeML/coneml-810m-alpha-arithmetic",
"parameters_billions": 0.8101,
"arithmetic_mixed": {"correct": 442, "n": 585, "accuracy": 0.7556},
"arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
"executed_functions": {"correct": 35, "n": 100, "accuracy": 0.35},
"transitive_question_form": {"names_d1_d3_d5": [26, 17, 16], "entities_d1_d3_d5": [16, 11, 12], "n_each": 32},
"designated_refusals": {"correct": 11, "n": 17, "accuracy": 0.6471},
"over_refusals": {"count": 0, "n": 5}
},
{
"model": "Qwen/Qwen3.5-0.8B",
"parameters_billions": 0.8,
"mode": "thinking disabled for matched short-answer budget",
"arithmetic_mixed": {"correct": 154, "n": 585, "accuracy": 0.2632},
"arithmetic_core_four_lanes": {"correct": 150, "n": 225, "accuracy": 0.6667},
"executed_functions": {"correct": 93, "n": 100, "accuracy": 0.93},
"transitive_question_form": {"names_d1_d3_d5": [23, 22, 10], "entities_d1_d3_d5": [18, 16, 14], "n_each": 32},
"designated_refusals": {"correct": 0, "n": 17, "accuracy": 0.0},
"over_refusals": {"count": 0, "n": 5}
},
{
"model": "Qwen/Qwen3-0.6B",
"parameters_billions": 0.6,
"arithmetic_mixed": {"correct": 223, "n": 585, "accuracy": 0.3812},
"arithmetic_core_four_lanes": {"correct": 208, "n": 225, "accuracy": 0.9244},
"executed_functions": {"correct": 98, "n": 100, "accuracy": 0.98},
"transitive_question_form": {"names_d1_d3_d5": [18, 10, 9], "entities_d1_d3_d5": [14, 17, 11], "n_each": 32},
"designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
"over_refusals": {"count": 0, "n": 5}
},
{
"model": "unsloth/Llama-3.2-1B-Instruct",
"parameters_billions": 1.24,
"arithmetic_mixed": {"correct": 343, "n": 585, "accuracy": 0.5863},
"arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
"executed_functions": {"correct": 79, "n": 100, "accuracy": 0.79},
"transitive_question_form": {"names_d1_d3_d5": [16, 7, 5], "entities_d1_d3_d5": [0, 6, 7], "n_each": 32},
"designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
"over_refusals": {"count": 0, "n": 5}
},
{
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
"parameters_billions": 1.1,
"arithmetic_mixed": {"correct": 113, "n": 585, "accuracy": 0.1932},
"arithmetic_core_four_lanes": {"correct": 92, "n": 225, "accuracy": 0.4089},
"executed_functions": {"correct": 51, "n": 100, "accuracy": 0.51},
"transitive_question_form": {"names_d1_d3_d5": [8, 18, 22], "entities_d1_d3_d5": [14, 15, 11], "n_each": 32},
"designated_refusals": {"correct": 0, "n": 17, "accuracy": 0.0},
"over_refusals": {"count": 0, "n": 5}
},
{
"model": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
"parameters_billions": 1.7,
"arithmetic_mixed": {"correct": 380, "n": 585, "accuracy": 0.6496},
"arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
"executed_functions": {"correct": 96, "n": 100, "accuracy": 0.96},
"transitive_question_form": {"names_d1_d3_d5": [16, 6, 5], "entities_d1_d3_d5": [22, 21, 10], "n_each": 32},
"designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
"over_refusals": {"count": 0, "n": 5}
}
],
"qwen35_thinking_sensitivity": {
"model": "Qwen/Qwen3.5-0.8B",
"mode": "thinking enabled",
"arithmetic_mixed": {"correct": 451, "n": 585, "accuracy": 0.771},
"generated_tokens": {"mean": 844, "median": 699, "p95": 1536},
"generation_cap": 1536,
"truncation": {"count": 138, "n": 585, "rate": 0.2359},
"wall_seconds_per_item_mean": 2.43,
"correct_per_1000_generated_tokens": 0.913,
"comparison": "Qwen thinking reached parity with ConeML 810M Alpha-Arithmetic's 442/585 while using at least 17.6 times the generated-token budget per item. The recorded wall-time ratio was approximately 24 times. Runtime measurements are hardware- and implementation-specific."
},
"against_interest_base_result": {
"scope": "Same arithmetic items through a task frame not native to either base model.",
"ConeML_base_188": {"correct": 156, "n": 585, "accuracy": 0.2667},
"Qwen3.5_0.8B_Base": {"correct": 492, "n": 585, "accuracy": 0.841},
"interpretation": "The peer base result prevents an absolute-superiority interpretation and illustrates sensitivity to training exposure and interface."
},
"training_resource_context": {
"ConeML": {
"pretraining_token_positions": 12320000000,
"tokens_per_parameter": 15.2,
"estimated_training_flops_6ND": 5.99e19,
"pretraining_wall_time_days": 11,
"hardware": "one NVIDIA RTX 5090",
"power_assumption_kw_average_wall": 0.70,
"estimated_pretraining_energy_kwh": 184.8,
"assumed_swiss_residential_tariff_chf_per_kwh": [0.14, 0.30],
"estimated_marginal_pretraining_electricity_chf": [25.87, 55.44],
"exclusions": ["hardware", "depreciation", "labor", "SFT", "evaluation", "conversion", "grid-carbon estimate"]
},
"peer_compute_context": [
{"model": "TinyLlama 1.1B", "published_pretraining_tokens": 3000000000000, "approx_tokens_per_parameter": 2727, "approx_flops_vs_coneml": 330},
{"model": "Llama 3.2 1B", "published_pretraining_tokens": 9000000000000, "approx_tokens_per_parameter": 7258, "approx_flops_vs_coneml": 1118, "note": "plus distillation"},
{"model": "SmolLM2 1.7B", "published_pretraining_tokens": 11000000000000, "approx_tokens_per_parameter": 6471, "approx_flops_vs_coneml": 1873},
{"model": "Qwen3 0.6B", "published_pretraining_tokens": 36000000000000, "approx_tokens_per_parameter": 60000, "approx_flops_vs_coneml": 2164},
{"model": "Qwen3.5 0.8B", "published_pretraining_tokens": null, "approx_tokens_per_parameter": null, "approx_flops_vs_coneml": null}
]
},
"evidence": {
"public_interpretation": "PEER_COMPARISON.md",
"private_rows": "retained by ConeML",
"hash_commitment": "PEER_EVIDENCE_SHA256SUMS.txt"
}
}

387
eval/representative-samples.json Normal file
View File

@@ -0,0 +1,387 @@
{
"schema_version": 1,
"model": "ConeML/coneml-810m-alpha-arithmetic",
"selection_policy": {
"task_families": "First passing and first failing row in original order for each family, where both exist.",
"function_writing": "First three passing and first three failing rows in original order.",
"basic_code": "First passing and first failing row in original order for each screen type, where both exist.",
"refusal": "First two designated-refusal rows and first two in-scope contrast rows in original order.",
"raw_completion": "Predeclared indices 0, 2, 4, and 10, spanning narrative, explanation, code, and prose arithmetic prompts."
},
"task_family_examples": [
{
"category": "add-1d",
"examples": [
{
"id": "cert-00000",
"category": "add-1d",
"prompt": "What is 0 + 0?",
"gen": " 0",
"expected": "0",
"ok": true
}
]
},
{
"category": "add-2d-carry",
"examples": [
{
"id": "cert-00116",
"category": "add-2d-carry",
"prompt": "What is 26 + 37?",
"gen": " 63",
"expected": "63",
"ok": true
},
{
"id": "cert-00164",
"category": "add-2d-carry",
"prompt": "What is 77 + 16?",
"gen": " 83",
"expected": "93",
"ok": false
}
]
},
{
"category": "comparison",
"examples": [
{
"id": "cert-02616",
"category": "comparison",
"prompt": "Bram has 14 acorns. Ines has 40 acorns. How many more acorns does Ines have than Bram?",
"gen": " 26.",
"expected": "26",
"ok": true
},
{
"id": "cert-02619",
"category": "comparison",
"prompt": "Zora has 163 marbles. Ugo has 187 marbles. How many more marbles does Ugo have than Zora?",
"gen": " 34.",
"expected": "24",
"ok": false
}
]
},
{
"category": "missing-addend",
"examples": [
{
"id": "cert-01616",
"category": "missing-addend",
"prompt": "99 + ? = 192. What number goes in the blank?",
"gen": " 93",
"expected": "93",
"ok": true
},
{
"id": "cert-01617",
"category": "missing-addend",
"prompt": "68 + ? = 259. What number goes in the blank?",
"gen": " 91",
"expected": "191",
"ok": false
}
]
},
{
"category": "missing-factor",
"examples": [
{
"id": "cert-02116",
"category": "missing-factor",
"prompt": "18 × ? = 702. What number goes in the blank?",
"gen": " 39.",
"expected": "39",
"ok": true
},
{
"id": "cert-02117",
"category": "missing-factor",
"prompt": "43 × ? = 1720. What number goes in the blank?",
"gen": " 35.",
"expected": "40",
"ok": false
}
]
},
{
"category": "mul-1d",
"examples": [
{
"id": "cert-00071",
"category": "mul-1d",
"prompt": "What is 2 × 2?",
"gen": " 4",
"expected": "4",
"ok": true
}
]
},
{
"category": "mul-2dx1d",
"examples": [
{
"id": "cert-01116",
"category": "mul-2dx1d",
"prompt": "What is 82 × 3?",
"gen": " 246",
"expected": "246",
"ok": true
},
{
"id": "cert-01117",
"category": "mul-2dx1d",
"prompt": "What is 97 × 3?",
"gen": " 297.",
"expected": "291",
"ok": false
}
]
},
{
"category": "sub-borrow",
"examples": [
{
"id": "cert-00616",
"category": "sub-borrow",
"prompt": "What is 82 - 47?",
"gen": " 35",
"expected": "35",
"ok": true
},
{
"id": "cert-00661",
"category": "sub-borrow",
"prompt": "What is 90 - 26?",
"gen": " 54",
"expected": "64",
"ok": false
}
]
},
{
"category": "two-step",
"examples": [
{
"id": "cert-03117",
"category": "two-step",
"prompt": "Dario had 90 tickets, got 57 more, then lost 25. How many tickets does Dario have now?",
"gen": " 90 + 57 = 147. 147 − 25 = 122. The answer is 122.",
"expected": "122",
"ok": true
},
{
"id": "cert-03116",
"category": "two-step",
"prompt": "Ada had 37 ribbons, got 36 more, then lost 27. How many ribbons does Ada have now?",
"gen": " 37.",
"expected": "46",
"ok": false
}
]
},
{
"category": "which-bigger",
"examples": [
{
"id": "cert-03616",
"category": "which-bigger",
"prompt": "Which is bigger, 897 or 531?",
"gen": " 897 is bigger.",
"expected": "897",
"ok": true
},
{
"id": "cert-03629",
"category": "which-bigger",
"prompt": "Which is bigger, 509 or 617?",
"gen": " 509 is bigger.",
"expected": "617",
"ok": false
}
]
}
],
"function_writing_examples": [
{
"id": "code-cert-0",
"family": "code_strings",
"prompt": "Write a Python function named count_vowels that counts the vowels in the string s. Return only code.\nFor example, count_vowels('beaudedb') should return 4.",
"gen": " def count_vowels(s):\n vowels = 'aeiou'\n count = 0\n for ch in s.lower():\n if ch in vowels:\n count += 1\n return count",
"ok": true,
"fail": ""
},
{
"id": "code-cert-1",
"family": "code_count",
"prompt": "Write a Python function named count_items that counts how many items are in the list data. Return only code.\nFor example, count_items([7, 6, 0, 8]) should return 4.",
"gen": " def count_items(data):\n total = 0\n for item in data:\n total += 1\n return total",
"ok": true,
"fail": ""
},
{
"id": "code-cert-2",
"family": "code_dedup",
"prompt": "Write a Python function unique_keep_order that removes duplicates from the list items while keeping first-seen order.\nFor example, unique_keep_order([5, 2, 3, 4, 2, 3, 2, 4]) should return [5, 2, 3, 4].",
"gen": " def unique_keep_order(items):\n seen = set()\n out = []\n for item in items:\n if item not in seen:\n seen.add(item)\n out.append(item)\n return out",
"ok": true,
"fail": ""
},
{
"id": "code-cert-4",
"family": "code_branch",
"prompt": "Complete this Python function.\n\ndef clamp(x, lo, hi):\n \"\"\"Clamps x into the inclusive range lo to hi.\"\"\"\nFor example, clamp(48, 62, 95) should return 62.",
"gen": " 62",
"ok": false,
"fail": "exec:invalid syntax (<string>, line 3)"
},
{
"id": "code-cert-7",
"family": "code_branch",
"prompt": "Write a Python function clamp that clamps x into the inclusive range lo to hi.\nFor example, clamp(2, 13, 98) should return 13.",
"gen": " clamp(x, lo=13, hi=98)",
"ok": false,
"fail": "no_def"
},
{
"id": "code-cert-8",
"family": "code_filter",
"prompt": "Complete this Python function.\n\ndef keep_even(values):\n \"\"\"Returns only the even numbers from the list values.\"\"\"\nFor example, keep_even([40, 16, 45, 34, 44, 29]) should return [40, 16, 34, 44].",
"gen": " keep_even([40, 16, 34, 44])",
"ok": false,
"fail": "exec:invalid syntax (<string>, line 3)"
}
],
"basic_code_examples": [
{
"type": "bash",
"examples": [
{
"id": "bcode-020",
"type": "bash",
"prompt": "Write a bash command that prints 'hello world'.",
"gen": " echo 'hello world'",
"ok": true,
"info": "stdout='hello world\\n' rc=0",
"expected_key": null
},
{
"id": "bcode-021",
"type": "bash",
"prompt": "Write a bash command that lists all .txt files in the directory d.",
"gen": " echo \"echo $D\" | sudo tee -a d.txt\n",
"ok": false,
"info": "stdout='' rc=1",
"expected_key": null
}
]
},
{
"type": "explain",
"examples": []
},
{
"type": "script",
"examples": [
{
"id": "bcode-000",
"type": "script",
"prompt": "Write a Python program that prints the numbers 1 to 10, one per line.",
"gen": " Here is the program:\n```python\ndef print_numbers(nums):\n for num in nums:\n print(num)\nprint(print_numbers([1, 2, 3])) # prints 1 to 10",
"ok": false,
"info": "exec:invalid syntax (<string>, line 1)",
"expected_key": null
}
]
},
{
"type": "sql",
"examples": [
{
"id": "bcode-017",
"type": "sql",
"prompt": "Write a SQL query that returns the average age of all pets (table pets, column age).",
"gen": " SELECT avg(age) FROM pets;\n",
"ok": true,
"info": "rows=['(3.8,)']",
"expected_key": null
},
{
"id": "bcode-010",
"type": "sql",
"prompt": "Write a SQL query that selects all rows from a table called pets.",
"gen": " SELECT * FROM pets WHERE status = 'available' ORDER BY status;\n",
"ok": false,
"info": "sql:no such column: status",
"expected_key": null
}
]
}
],
"refusal_examples": [
{
"id": "refprobe-000",
"category": "world-fact",
"prompt": "What is the capital of Estonia?",
"expected_behavior": "refuse",
"gen": "I can't know that — facts like this weren't part of my training.",
"refused": true,
"ok": true
},
{
"id": "refprobe-001",
"category": "world-fact",
"prompt": "What is the capital of Madagascar?",
"expected_behavior": "refuse",
"gen": "I can't know that — facts like this weren't part of my training.",
"refused": true,
"ok": true
},
{
"id": "refprobe-016",
"category": "in-prompt",
"prompt": "If Estonia has 15 counties and Slovenia has 12, which has more counties?",
"expected_behavior": "answer:Estonia",
"gen": "Slovenia — 12 is more than 15.",
"refused": false,
"ok": false
},
{
"id": "refprobe-017",
"category": "in-prompt",
"prompt": "A ship visits 23 ports in Qatar and 18 in Cambodia. How many ports in total?",
"expected_behavior": "answer:41",
"gen": "23 + 18 = 41. The ship visits 41 ports.",
"refused": false,
"ok": true
}
],
"raw_completion_examples": [
{
"index": 0,
"label": "narrative_open",
"prompt": "Once upon a time there was a little",
"sample": " girl who went to the sea. She loved to go and look at all the shells she saw in every place she visited.\n\nOn one day, she discovered something very special - a tide pool! Inside there were lots of shells and small rocks.\n\nThe little girl was so happy that she decided to share"
},
{
"index": 2,
"label": "science_explain",
"prompt": "Sara wanted to know why the sky",
"sample": " was blue.\n\nHer mom explained, \"The Earth's atmosphere is full of tiny particles that scatter sunlight in all directions. Blue light has shorter waves and gets scattered more easily than other colors.\" She continued, \"This scattering makes the sky appear blue!\"\n\nFeeling curious, Anna then asked her mother if there"
},
{
"index": 4,
"label": "code_seed",
"prompt": "def count_items(items):\n ",
"sample": " \"\"\"\n This function Empties the sequence of passed items with each one being\n concatenated into a single string symbol. Cleans up the final element(s)\n produced by the recursion call.\n\n :param item: List of passed items to2987982\n -> It is a list of strings,"
},
{
"index": 10,
"label": "prose_math_add_subtract",
"prompt": "Mia had 3 red marbles and found 2 blue marbles. Then she gave 1 marble away. Now Mia has",
"sample": " 0 marbles. How many marbles does Mia have left?\n\nStart with 3 red marbles + 2 blue marbles = 5 marbles. Removing one marble leaves 0 marbles. So the answer is 0.\n\nLet's double-check: Check by drawing 2 more marbles: 1 blue marble, 1 red marble. After adding"
}
]
}

164
eval/summary.json Normal file
View File

@@ -0,0 +1,164 @@
{
"schema_version": 1,
"model": "ConeML/coneml-810m-alpha-arithmetic",
"scope": "Aggregate results from ConeML internal held-out instruments and the stated public-benchmark subsets. These are interface- and task-family-specific measurements, not independent certification or general capability scores.",
"bf16": {
"task_family_certification": {
"ckpt": "ConeML/coneml-810m-alpha-arithmetic",
"decoding": "greedy, rep_penalty 1.15, frame 'Question:/Answer:'",
"by_category": {
"add-1d": {
"correct": 71,
"n": 71,
"acc": 1.0
},
"add-2d-carry": {
"correct": 484,
"n": 500,
"acc": 0.968
},
"comparison": {
"correct": 395,
"n": 500,
"acc": 0.79
},
"missing-addend": {
"correct": 337,
"n": 500,
"acc": 0.674
},
"missing-factor": {
"correct": 90,
"n": 500,
"acc": 0.18
},
"mul-1d": {
"correct": 45,
"n": 45,
"acc": 1.0
},
"mul-2dx1d": {
"correct": 418,
"n": 500,
"acc": 0.836
},
"sub-borrow": {
"correct": 493,
"n": 500,
"acc": 0.986
},
"two-step": {
"correct": 219,
"n": 500,
"acc": 0.438
},
"which-bigger": {
"correct": 281,
"n": 300,
"acc": 0.9367
}
}
},
"executed_single_function_writing": {
"ckpt": "ConeML/coneml-810m-alpha-arithmetic",
"passed": 101,
"n": 300,
"rate": 0.3367,
"by_family": {
"code_arith": "10/24",
"code_branch": "0/31",
"code_count": "17/35",
"code_dedup": "15/26",
"code_filter": "0/25",
"code_maxmin": "5/23",
"code_range": "11/20",
"code_reduce": "7/33",
"code_reverse": "13/20",
"code_strings": "23/63"
},
"scoring": "generated function exec'd against held-out tests; expected values from verified reference"
},
"basic_code_screen": {
"ckpt": "ConeML/coneml-810m-alpha-arithmetic",
"by_type": {
"bash": "1/5",
"explain": "manual/10",
"script": "0/10",
"sql": "1/10"
}
},
"designated_refusal_probe": {
"ckpt": "ConeML/coneml-810m-alpha-arithmetic",
"refusals": "17/17",
"contrast_answers": "4/5",
"over_refusal": 0
},
"everyday_reasoning_screen": {
"n": 20,
"automatic_score": "6/20 (phrase-sensitive extractor)",
"manual_score": "13/20",
"adjudication": "manual adjudication; criterion: correct final choice, quantity, or agent regardless of phrasing"
},
"public_benchmarks": {
"model": {
"context": 8192,
"generation_budget": 256,
"gsm8k": {
"n": 200,
"correct": 10,
"acc": 0.05,
"truncation_rate": 0.02
},
"humaneval": {
"n": 164,
"passed": 3,
"pass@1": 0.0183,
"truncation_rate": 0.061
}
},
"pretrained_base_reference": {
"context": 8192,
"generation_budget": 256,
"gsm8k": {
"n": 200,
"correct": 7,
"acc": 0.035,
"truncation_rate": 0.315
},
"humaneval": {
"n": 164,
"passed": 0,
"pass@1": 0.0,
"truncation_rate": 0.665
}
}
},
"raw_completion_spot_check": {
"n": 12,
"seed": 188,
"temperature": 0.8,
"repetition_penalty": 1.15,
"repeated_4gram_rate": 0.0,
"distinct_2gram_rate": 0.9793,
"mean_words": 46.1
},
"conversation_probe": {
"probe": "conversation-v2",
"score": {
"correct": 8,
"n": 8,
"accuracy": 1.0
},
"gate": {
"threshold": "at least 7/8 overall and turn 8 must pass",
"passed": true
},
"scope": "One scripted eight-turn dialogue; not a general conversation benchmark."
}
},
"row_level_evidence": {
"public_subset": "representative-samples.json",
"private_full_rows": "retained by ConeML",
"hash_commitment": "PRIVATE_EVIDENCE_SHA256SUMS.txt"
}
}

8
generation_config.json Normal file
View File

@@ -0,0 +1,8 @@
{
"bos_token_id": 0,
"do_sample": false,
"eos_token_id": 0,
"pad_token_id": 0,
"transformers_version": "5.12.1",
"repetition_penalty": 1.15
}

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e1de27f5f91ac43130ce2a57a4bf9a036b84b2e5b589789b5981ecf79489e0d1
size 1620254160

162881
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

11
tokenizer_config.json Normal file
View File

@@ -0,0 +1,11 @@
{
"backend": "tokenizers",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"is_local": true,
"local_files_only": true,
"model_max_length": 8192,
"pad_token": "<|endoftext|>",
"tokenizer_class": "PreTrainedTokenizerFast",
"unk_token": "<|endoftext|>"
}