270 lines
11 KiB
Markdown
270 lines
11 KiB
Markdown
---
|
||
license: cc-by-nc-4.0
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
language:
|
||
- en
|
||
---
|
||
|
||
# ConeML 810M Alpha-Arithmetic
|
||
|
||
ConeML 810M Alpha-Arithmetic is an 810,112,512-parameter, decoder-only
|
||
language model trained from scratch and instruction-tuned for a narrow
|
||
small-number arithmetic profile. It shares its pretrained base with
|
||
`coneml-810m-alpha` and differs in fine-tuning composition.
|
||
|
||
This variant records the stronger carry/borrow profile of the pair. The
|
||
companion `coneml-810m-alpha` has substantially stronger function-writing
|
||
and two-step results. Results are specific to the documented prompt
|
||
formats and task families, not claims of general mathematical, coding,
|
||
reasoning, or factual-knowledge performance.
|
||
|
||
- Architecture: LLaMA-style; 28 layers, d=1536, 12 heads (3 KV), tied
|
||
embeddings, 8192 context, 32,768-token BPE tokenizer
|
||
- Pretraining corpus: 27.87B curated tokens
|
||
- Selected pretrained-base exposure: approximately 12.32B token
|
||
positions, or 15.2 tokens per parameter
|
||
- Pretraining wall time: approximately 11 days on one NVIDIA RTX 5090;
|
||
fine-tuning and evaluation are excluded
|
||
- Release `model.safetensors` SHA-256:
|
||
`e1de27f5f91ac43130ce2a57a4bf9a036b84b2e5b589789b5981ecf79489e0d1`
|
||
|
||
## 1. Measured profile
|
||
|
||
Across four held-out core lane sets—1-digit addition, 1-digit
|
||
multiplication, carry addition, and borrow subtraction—the model scores
|
||
1,093/1,116 (97.9%). Two of those four held-out sets scored 100%.
|
||
|
||
This is a result on four explicitly named, task-family-matched sets. It is
|
||
not a claim of 97.9% general arithmetic accuracy. ConeML internal
|
||
certification is not independent third-party certification.
|
||
|
||
| held-out task family | result | n |
|
||
|---|---:|---:|
|
||
| 2–3-digit addition with carry | 96.8% | 500 |
|
||
| 2-digit subtraction with borrow | 98.6% | 500 |
|
||
| 1-digit addition, including 19 zero-operand items | 100% | 71 |
|
||
| 1-digit multiplication | 100% | 45 |
|
||
| "Which is bigger, X or Y?" | 93.7% | 300 |
|
||
| comparison word problems | 79.0% | 500 |
|
||
| missing addend | 67.4% | 500 |
|
||
| two-step word problems | 43.8% | 500 |
|
||
| 2-digit × 1-digit multiplication | 83.6% | 500 |
|
||
| missing factor | 18.0% | 500 |
|
||
| Python function writing (single function, executed) | 33.7% | 300 |
|
||
|
||
Behavioral screens:
|
||
|
||
- Designated-refusal probe: 17/17 refusal prompts produced refusals, and
|
||
4/5 in-scope contrast prompts were answered correctly. The miss was an
|
||
incorrect comparison answer, not an over-refusal. This is a small
|
||
targeted policy test, not general epistemic calibration.
|
||
- Conversation-v2: 8/8 turns passed in one scripted dialogue covering
|
||
cross-turn arithmetic, story revision, comparison explanation, and one
|
||
unknown-personal-fact refusal. This is a single dialogue, not a general
|
||
conversation benchmark.
|
||
- Everyday-reasoning screen: 13/20 by manual adjudication; the
|
||
phrase-sensitive automatic scorer recorded 6/20.
|
||
|
||
## 2. Peer comparison and efficiency
|
||
|
||
The pair was tested against five post-trained peers on the same reduced
|
||
diagnostic screen, through each model's native instruction interface,
|
||
with greedy decoding, repetition penalty 1.15, and matched short-answer
|
||
budgets.
|
||
|
||
| model | parameters | mixed arithmetic (n=585) | executed functions (n=100) | transitive names d1/d3/d5 (n=32 each) | designated refusals (n=17) |
|
||
|---|---:|---:|---:|---:|---:|
|
||
| ConeML 810M Alpha | 0.81B | 421 (72.0%) | 83 | 23/15/15 | **13** |
|
||
| ConeML 810M Alpha-Arithmetic | 0.81B | **442 (75.6%)** | 35 | 26/17/16 | 11 |
|
||
| Qwen3.5 | 0.8B | 154 (26.3%) | 93 | 23/22/10 | 0 |
|
||
| Qwen3 | 0.6B | 223 (38.1%) | **98** | 18/10/9 | 1 |
|
||
| Llama 3.2 Instruct | 1.24B | 343 (58.6%) | 79 | 16/7/5 | 1 |
|
||
| TinyLlama Chat | 1.1B | 113 (19.3%) | 51 | 8/18/22 | 0 |
|
||
| SmolLM2 Instruct | 1.7B | 380 (65.0%) | 96 | 16/6/5 | 1 |
|
||
|
||
This is not a general leaderboard: the task families match ConeML's
|
||
trained surfaces. On this screen, the ConeML pair led mixed arithmetic;
|
||
ConeML 810M Alpha exceeded Llama 3.2 and TinyLlama on function writing
|
||
but trailed Qwen3.5, Qwen3, and SmolLM2. All models recorded zero
|
||
over-refusals on five in-scope contrasts.
|
||
|
||
On the question-form name-chain depth screen, ConeML 810M
|
||
Alpha-Arithmetic recorded 26/32, 17/32, and 16/32 at depths 1, 3, and 5;
|
||
ConeML 810M Alpha recorded 23/32, 15/32, and 15/32. The arithmetic
|
||
variant exceeded Qwen3 and Llama 3.2 at all three shown depths and
|
||
Qwen3.5 at depths 1 and 5, while Qwen3.5 led it at depth 3 and TinyLlama
|
||
led the group at depth 5. Entity-chain controls were mixed and are
|
||
reported in the full table. Chance is 1/(depth+1); this is an
|
||
exact-selection surface test, not a claim of general reasoning.
|
||
|
||
Qwen3.5 with thinking enabled reached 451/585 (77.1%), compared with
|
||
this release's 442/585 (75.6%), while using at least 17.6 times the
|
||
generated-token budget per item and approximately 24 times the recorded
|
||
wall time. Runtime ratios are implementation-specific.
|
||
|
||
ConeML's selected base consumed 12.32B token positions—15.2 tokens per
|
||
parameter and approximately 5.99e19 training FLOPs under the `6ND`
|
||
convention. Pretraining took approximately 11 days on one RTX 5090.
|
||
Assuming, rather than claiming to have metered, a 0.70 kW average wall
|
||
draw gives 184.8 kWh; at CHF 0.14–0.30/kWh, that is approximately
|
||
CHF 26–55 of marginal pretraining electricity. This excludes hardware,
|
||
labor, SFT, evaluation, conversion, and emissions accounting.
|
||
|
||
The complete category table, peer wins, adverse base result, evaluated
|
||
revisions, assumptions, and frozen evidence commitments are in
|
||
[`eval/PEER_COMPARISON.md`](eval/PEER_COMPARISON.md).
|
||
|
||
## 3. Standard benchmark limits
|
||
|
||
Zero-shot, greedy decoding with a 256-token generation cap. GSM8K uses
|
||
strict final-number matching; HumanEval reports pass@1 from executed
|
||
tests.
|
||
|
||
| benchmark | this model | same base, no fine-tune |
|
||
|---|---:|---:|
|
||
| GSM8K (test, first 200) | 10/200 (5.0%) | 7/200 (3.5%) |
|
||
| HumanEval (pass@1, tests executed) | 3/164 (1.8%) | 0/164 (0.0%) |
|
||
|
||
These results are weak and define the boundary of the specialization:
|
||
the four-lane arithmetic profile does not transfer to broad GSM8K or
|
||
HumanEval performance.
|
||
|
||
## 4. Supported prompt formats
|
||
|
||
Evaluated decoding is greedy with repetition penalty 1.15, as pinned in
|
||
`generation_config.json`.
|
||
|
||
```text
|
||
# task format (single turn)
|
||
Question: <task>
|
||
Answer:
|
||
|
||
# dialogue format (chat_template.jinja)
|
||
User:
|
||
<message>
|
||
Assistant:
|
||
```
|
||
|
||
Dialogue generation ends at `<|endoftext|>`. Prompting outside these
|
||
formats, including paraphrases, was not systematically evaluated except
|
||
for the published raw-completion spot check.
|
||
|
||
## 5. Quantized variants
|
||
|
||
Q8_0 GGUF was evaluated separately under llama.cpp. Its runtime and stop
|
||
handling differ from the bf16 harness, so score differences cannot be
|
||
attributed solely to quantization.
|
||
|
||
| held-out task family | Q8_0 |
|
||
|---|---:|
|
||
| subtraction with borrow | 98.6% (n=500) |
|
||
| addition with carry | 92.6% (n=500) |
|
||
| 1-digit multiplication | 100% (n=45) |
|
||
| two-step word problems | 32.6% (n=500) |
|
||
|
||
For local GGUF inference:
|
||
|
||
```bash
|
||
llama-cli \
|
||
-m coneml-810m-alpha-arithmetic-Q8_0.gguf \
|
||
-p $'Question: What is 47 + 38?\nAnswer:' \
|
||
-n 32 --repeat-penalty 1.15 --no-conversation --single-turn
|
||
```
|
||
|
||
For Ollama, download `Modelfile` beside the Q8_0 file, then run:
|
||
|
||
```bash
|
||
ollama create coneml-810m-alpha-arithmetic -f Modelfile
|
||
ollama run coneml-810m-alpha-arithmetic "What is 47 + 38?"
|
||
```
|
||
|
||
## 6. Loading with Transformers
|
||
|
||
```python
|
||
import torch
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
repo_id = "ConeML/coneml-810m-alpha-arithmetic"
|
||
tokenizer = AutoTokenizer.from_pretrained(repo_id)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
repo_id,
|
||
dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
)
|
||
|
||
prompt = "Question: What is 47 + 38?\nAnswer:"
|
||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||
output = model.generate(
|
||
**inputs,
|
||
max_new_tokens=32,
|
||
do_sample=False,
|
||
repetition_penalty=1.15,
|
||
)
|
||
answer = tokenizer.decode(
|
||
output[0][inputs["input_ids"].shape[-1]:],
|
||
skip_special_tokens=True,
|
||
)
|
||
print(answer)
|
||
```
|
||
|
||
This is a text-only causal language model. Load it with
|
||
`AutoModelForCausalLM` or `LlamaForCausalLM`, not a multimodal model class.
|
||
|
||
## 7. Intended use and safety
|
||
|
||
This release is intended for research and constrained local workflows
|
||
inside its measured small-number arithmetic envelope. It is not a general
|
||
mathematics system. Validate outputs whenever exactness matters.
|
||
|
||
It is not intended for factual retrieval, high-stakes medical, legal, or
|
||
financial decisions, autonomous action, or unsandboxed execution of
|
||
generated code. Generated code may be incomplete or unsafe; inspect and
|
||
test it in an isolated environment.
|
||
|
||
## 8. Known limitations
|
||
|
||
- The measured strength is narrow. Missing factor scored 18.0%, two-step
|
||
word problems 43.8%, 2-digit × 1-digit multiplication 83.6%, and
|
||
executed single-function writing 33.7%.
|
||
- Not a code model: complete programs scored 0/10, executable SQL 1/10,
|
||
and bash 1/5.
|
||
- The model was not evaluated as a factual-QA system and refused all six
|
||
real-world factual prompts in the designated-refusal probe.
|
||
- Money/decimal change-making scored 0/3 on the held-out screen; the model
|
||
also showed yes-bias on feasibility questions.
|
||
- Multi-step GSM-class reasoning remained weak.
|
||
- Frameless prompting was spot-checked, not systematically certified.
|
||
|
||
## 9. Evidence and reproducibility
|
||
|
||
The `eval/` directory publishes:
|
||
|
||
- `summary.json`: all aggregate bf16 results used above;
|
||
- `representative-samples.json`: deterministic, rule-selected examples
|
||
from ConeML-owned probes;
|
||
- `EVALUATION_METHODOLOGY.md`: evaluation definitions and scope;
|
||
- separate Q8_0 and focused aggregate results; and
|
||
- `PRIVATE_EVIDENCE_SHA256SUMS.txt`: commitments to the complete
|
||
row-level evidence retained by ConeML.
|
||
- `peer-comparison-summary.json`, `PEER_COMPARISON.md`, and
|
||
`PEER_EVIDENCE_SHA256SUMS.txt`: the peer results, interpretation, and
|
||
commitments to the retained peer-generation evidence.
|
||
|
||
Complete proprietary probe rows and generations are retained privately to
|
||
avoid releasing the full measurement instruments. The hashes permit later
|
||
disclosure to be checked against the evidence frozen at release time.
|
||
Public GSM8K and HumanEval rows are not redistributed; only aggregate
|
||
results are published. `SHA256SUMS.txt` covers every shipped file.
|
||
|
||
## 10. Research license and commercial work
|
||
|
||
Released for non-commercial use under CC BY-NC 4.0. Commercial use is not
|
||
granted by this release. See `LICENSE.md`.
|
||
|
||
For commercial evaluation, model-engineering, or licensing enquiries,
|
||
contact [contact@coneml.com](mailto:contact@coneml.com) or visit
|
||
[ConeML for Organizations](https://coneml.com/organizations/). Commercial availability does not
|
||
imply that this alpha checkpoint is suitable for an unvalidated
|
||
production workflow.
|