ConeML/coneml-348m-alpha-polish900
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: ConeML/coneml-348m-alpha-polish900
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-16 09:05:19 +08:00
commit a0dc46947c
15 changed files with 169755 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

174
README.md Normal file
View File

@@ -0,0 +1,174 @@
---
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: text-generation
tags:
- text-generation
- llama
- scratch-trained
- small-language-model
- research-artifact
---
# ConeML 348M Alpha Polish900
ConeML 348M Alpha Polish900 is a 348M-parameter scratch-trained alpha model from a custom layered curriculum, followed by staged SFT activation. This release is a research artifact and alpha candidate, not a polished general assistant.
The main result is format activation: raw-completion probes understated the base checkpoint, while the trained chat format activated transitive binding and simple code-body generation. Arithmetic remains unresolved.
## Why ConeML Exists
ConeML is an independent research effort testing whether compact language models can be built from scratch through deliberately staged curricula rather than scale alone.
The central question is whether small models can develop usable reasoning substrate through corpus design, curriculum order, and staged activation training. In v10, the clearest signal was transitive relation binding for name-like entities. In raw base probes this was near-absent; after focused SFT it reached 100% on a fixed-template internal chat probe (depths 1-3, N=128 per depth). A held-out probe (2026-06-23, N=128 per depth, depths 1-5) confirms this generalizes across new names and new relation wording: with held-out names and a new older/younger relation, chat first-choice accuracy was 79% / 89% / 88% / 77% / 71% across depths 1-5, well above chance. Generalization is weaker under unseen query phrasing (56% / 73% / 59% / 48% / 34%) and falls to roughly chance for non-name entities such as colored cards (51% / 50% / 41% / 31% / 28% vs chance 50% / 33% / 25% / 20% / 17%). The result is real and held-out, but the binding is name-shaped and surface-sensitive, not general abstract transitive reasoning.
`coneml-348m-alpha-polish900` is the first public artifact from that work. Its strongest result is not that every capability is solved, but that raw completion understated parts of the model: transitive reasoning and simple code-body behavior became much more visible after targeted SFT, while arithmetic remained a real unresolved weakness.
## Intended Format
Use the role-marker chat format:
```text
User:
<instruction>
Assistant:
```
Raw completion is not the intended use surface for the tuned checkpoint.
## License
Released for non-commercial use under CC BY-NC 4.0. Commercial use is not granted by this release.
## Loading
**This is a text-only causal language model. Use `AutoModelForCausalLM` or `LlamaForCausalLM`, not a multimodal model class.**
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "ConeML/coneml-348m-alpha-polish900"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
torch_dtype=torch.float32,
device_map="auto",
)
prompt = "User:\nWhat is 2 + 3? Return only the number.\nAssistant:\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=32,
do_sample=False,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
```
## Architecture
- Family: Llama-style decoder
- Parameters: approximately 348M
- Layers: 30
- Hidden size: 1024
- Attention heads: 8
- KV heads: 2
- Vocab size: 32768
- Context length: 512
- RoPE theta: 1000000
- Tokenizer: custom 32K tokenizer
## Training Lineage
- Base selected near the balanced pretrain region: `ckpt_0210000.pt`
- SFT stages: base210 -> SFT300 -> focus600 -> polish900
- Final exported checkpoint: `runs/v10_348m_cone_sft_polish900/sft_ckpt_0000300.pt`
## Internal Probe Results
These are internal diagnostic probes, not public benchmark claims.
### Activation Progression
Raw-completion transitive first-name accuracy improved gradually across SFT stages:
| Stage | Depth 1 | Depth 2 | Depth 3 |
|---|---:|---:|---:|
| Base210 raw | 32.03% | 24.22% | 5.47% |
| SFT300 raw | 32.81% | 35.94% | 12.50% |
| Focus600 raw | 46.88% | 61.72% | 43.75% |
| Polish900 raw | 53.91% | 64.84% | 53.12% |
Chat-format transitive binding reached 100% on the fixed-template Focus600 and Polish900 internal probes (depths 1-3, in-distribution name pool). A separate held-out probe (2026-06-23) confirms generalization to new names and a new relation wording (79-89% at depths 1-3) but shows near-chance performance on non-name entities; see Held-out Transitive Validation below.
| Stage | Depth 1 | Depth 2 | Depth 3 | Math final numeric | Code strict exec |
|---|---:|---:|---:|---:|---:|
| Focus600 chat | 100.00% | 100.00% | 100.00% | 28.12% | 0.00% |
| Polish900 chat | 100.00% | 100.00% | 100.00% | 35.94% | 16.67% |
### Chat-Format Probe
| Probe | Result |
|---|---:|
| Transitive depth 1 first-name | 100.00% |
| Transitive depth 2 first-name | 100.00% |
| Transitive depth 3 first-name | 100.00% |
| Math final numeric | 35.94% |
| Math answer-anywhere | 35.94% |
| Code strict exec | 16.67% |
Code note: strict execution is mostly limited by indentation. The Polish900 chat probe generated plausible correct return expressions for the 6 simple function probes, but 5/6 were emitted with the wrong leading whitespace; under indentation normalization, those return bodies execute.
### Raw-Completion Probe
| Probe | Result |
|---|---:|
| Transitive depth 1 first-name | 53.91% |
| Transitive depth 2 first-name | 64.84% |
| Transitive depth 3 first-name | 53.12% |
| Math final numeric | 17.97% |
| Code body rate | 0.00% |
### Held-out Transitive Validation (2026-06-23)
Polish900, chat surface, first-choice accuracy, N=128 per depth.
| Suite | D1 | D2 | D3 | D4 | D5 |
|---|---:|---:|---:|---:|---:|
| SFT template + held-out names | 94.5% | 96.1% | 94.5% | 94.5% | 82.8% |
| Held-out names + new relation (older/younger) | 78.9% | 89.1% | 88.3% | 76.6% | 71.1% |
| Unseen query phrasing + held-out names | 56.3% | 73.4% | 59.4% | 48.4% | 33.6% |
| Non-name entities (cards, comes before) | 50.8% | 50.0% | 41.4% | 30.5% | 28.1% |
| Chance | 50% | 33% | 25% | 20% | 17% |
Takeaway: held-out validation supports generalization across new names and relation wording for name-like entities in chat format. It is weaker under unseen query phrasing and at or near chance for non-name entities and deeper chains. Raw completion is weaker than chat in every suite.
## Strengths
- Scratch-trained 348M model from a custom layered curriculum.
- Strong SFT activation curve on transitive relation binding.
- Chat-format transitive relation binding reaches 100% on a fixed-template internal probe (depths 1-3) and is held-out validated for name-like entities (79-89% at depths 1-3 with new names and a new relation wording). It degrades under unseen query phrasing and drops to roughly chance for non-name entities, so it is name-shaped binding rather than general transitive reasoning.
- Simple code return bodies appear in chat format; the remaining failure is mostly indentation/formatting, not missing return-body content on the internal probe.
## Known Limitations
- Arithmetic remains the weakest major capability lane. Chat-format final numeric accuracy reached 35.94% on the internal probe, but reliable multi-digit arithmetic is not solved.
- Raw completion is poor for code bodies and is not the intended tuned interface.
- Code indentation is unstable without postprocessing.
- Internal probes only; this card makes no public benchmark claims.
- This is an alpha/research release, not a replacement for larger general assistants.
## Reproducibility Artifacts
This local release directory includes:
- `training_summary.json`
- `evals/v10_diagnostic_probe_0210000_gpu_matched_sft300.json`
- `evals/diag_postsft_sft300_gpu.json`
- `evals/diag_focus600_raw_gpu.json`
- `evals/chat_activation_focus600_gpu.json`
- `evals/chat_activation_polish900_gpu.json`
- `evals/diag_polish900_raw_gpu.json`

8
chat_template.jinja Normal file
View File

@@ -0,0 +1,8 @@
{% for message in messages %}{% if message['role'] == 'system' %}System:
{{ message['content'] }}
{% elif message['role'] == 'user' %}User:
{{ message['content'] }}
{% elif message['role'] == 'assistant' %}Assistant:
{{ message['content'] }}{% if not loop.last %}{{ eos_token }}
{% endif %}{% endif %}{% endfor %}{% if add_generation_prompt %}Assistant:
{% endif %}

33
config.json Normal file
View File

@@ -0,0 +1,33 @@
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 1,
"dtype": "float32",
"eos_token_id": 2,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 1024,
"initializer_range": 0.02,
"intermediate_size": 2560,
"max_position_embeddings": 512,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 8,
"num_hidden_layers": 30,
"num_key_value_heads": 2,
"pad_token_id": null,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_parameters": {
"rope_theta": 10000.0,
"rope_type": "default"
},
"rope_theta": 1000000,
"tie_word_embeddings": true,
"transformers_version": "5.5.4",
"use_cache": true,
"vocab_size": 32768
}

492
evals/chat_activation_focus600_gpu.json Normal file
View File

@@ -0,0 +1,492 @@
{
"chat_format": "User:\\n...\\nAssistant:\\n",
"ckpt": "runs/v10_348m_cone_sft_focus600/sft_ckpt_0000300.pt",
"code": {
"N": 6,
"ast_rate": 1.0,
"body_rate": 0.0,
"ci95_exec": [
0.0,
0.3903430336530645
],
"examples": [
{
"ast_ok": true,
"body_ok": false,
"code": "def add_one(n):\n \"\"\"Return n plus one.\"\"\"\n",
"exec_ok": false,
"generated": " return n + 1",
"name": "add_one",
"return_ok": false
},
{
"ast_ok": true,
"body_ok": false,
"code": "def reverse_string(s):\n \"\"\"Return s reversed.\"\"\"\n",
"exec_ok": false,
"generated": "return s[::-1]",
"name": "reverse_string",
"return_ok": false
},
{
"ast_ok": true,
"body_ok": false,
"code": "def count_vowels(s):\n \"\"\"Return the number of vowels in s.\"\"\"\n",
"exec_ok": false,
"generated": " return sum(1 for ch in s.lower() if ch in 'aeiou')",
"name": "count_vowels",
"return_ok": false
},
{
"ast_ok": true,
"body_ok": false,
"code": "def is_even(n):\n \"\"\"Return True if n is even.\"\"\"\n",
"exec_ok": false,
"generated": " return n % 2 == 0",
"name": "is_even",
"return_ok": false
},
{
"ast_ok": true,
"body_ok": false,
"code": "def sum_list(items):\n \"\"\"Return the sum of the numbers in items.\"\"\"\n",
"exec_ok": false,
"generated": " return sum(items)",
"name": "sum_list",
"return_ok": false
},
{
"ast_ok": true,
"body_ok": false,
"code": "def first_item(items):\n \"\"\"Return the first item in a non-empty list.\"\"\"\n",
"exec_ok": false,
"generated": " return items[0]",
"name": "first_item",
"return_ok": false
}
],
"exec_rate": 0.0,
"return_rate": 0.0
},
"device": "cuda",
"generated_at": "2026-06-22T20:54:24Z",
"math": {
"N": 128,
"answer_anywhere_rate": 0.28125,
"ci95_anywhere": [
0.2106120712149715,
0.36463580199045076
],
"examples": [
{
"anywhere": true,
"final": true,
"first": true,
"gen": " 6",
"gold": 6,
"prompt": "Sam has 4 apples. Lily has 2 more apples than Sam. How many apples does Lily have? Return only the number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 2",
"gold": 4,
"prompt": "Mia had 3 red marbles and found 2 blue marbles. Then she gave 1 marble away. Now Mia has how many marbles? Return only the number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 32",
"gold": 20,
"prompt": "What is 68 - 48? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 196",
"gold": 214,
"prompt": "What is 118 + 96? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 105",
"gold": 113,
"prompt": "What is 109 + 4? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 284",
"gold": 1110,
"prompt": "What is 74 * 15? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 103",
"gold": 55,
"prompt": "What is 97 - 42? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 75",
"gold": 51,
"prompt": "What is 102 - 51? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 18",
"gold": 2,
"prompt": "What is 22 - 20? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 54",
"gold": 58,
"prompt": "What is 26 + 32? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 365",
"gold": 1469,
"prompt": "What is 113 * 13? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 44",
"gold": 43,
"prompt": "What is 43 * 1? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 109",
"gold": 103,
"prompt": "What is 111 - 8? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 110",
"gold": 122,
"prompt": "What is 79 + 43? Return only the final number."
},
{
"anywhere": true,
"final": true,
"first": true,
"gen": " 120",
"gold": 120,
"prompt": "What is 60 * 2? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 155",
"gold": 151,
"prompt": "What is 72 + 79? Return only the final number."
}
],
"final_numeric_accuracy": 0.28125,
"first_numeric_accuracy": 0.28125
},
"step": 300,
"transitive": {
"depth_1": {
"N": 128,
"answer_anywhere_rate": 1.0,
"chance": 0.5,
"ci95_first": [
0.9708620041018919,
0.9999999999999999
],
"examples": [
{
"anywhere": true,
"first_name": "Dana",
"first_ok": true,
"gen": " Dana",
"gold": "Dana",
"prompt": "Dana is taller than Cara. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Cara",
"first_ok": true,
"gen": " Cara",
"gold": "Cara",
"prompt": "Dana is taller than Cara. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Dana",
"first_ok": true,
"gen": " Dana",
"gold": "Dana",
"prompt": "Dana is taller than Gina. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Gina",
"first_ok": true,
"gen": " Gina",
"gold": "Gina",
"prompt": "Dana is taller than Gina. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Dana",
"first_ok": true,
"gen": " Dana",
"gold": "Dana",
"prompt": "Dana is taller than Hugo. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Hugo",
"first_ok": true,
"gen": " Hugo",
"gold": "Hugo",
"prompt": "Dana is taller than Hugo. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Gina",
"first_ok": true,
"gen": " Gina",
"gold": "Gina",
"prompt": "Gina is taller than Dana. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Dana",
"first_ok": true,
"gen": " Dana",
"gold": "Dana",
"prompt": "Gina is taller than Dana. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Cara",
"first_ok": true,
"gen": " Cara",
"gold": "Cara",
"prompt": "Cara is taller than Finn. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Cara is taller than Finn. Of all of them, the shortest is who? Return only the name."
}
],
"first_name_accuracy": 1.0
},
"depth_2": {
"N": 128,
"answer_anywhere_rate": 1.0,
"chance": 0.3333333333333333,
"ci95_first": [
0.9708620041018919,
0.9999999999999999
],
"examples": [
{
"anywhere": true,
"first_name": "Evan",
"first_ok": true,
"gen": " Evan",
"gold": "Evan",
"prompt": "Evan is taller than Dana. Dana is taller than Anna. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Anna",
"first_ok": true,
"gen": " Anna",
"gold": "Anna",
"prompt": "Evan is taller than Dana. Dana is taller than Anna. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Ben",
"first_ok": true,
"gen": " Ben",
"gold": "Ben",
"prompt": "Ben is taller than Evan. Evan is taller than Anna. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Anna",
"first_ok": true,
"gen": " Anna",
"gold": "Anna",
"prompt": "Ben is taller than Evan. Evan is taller than Anna. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Finn is taller than Hugo. Hugo is taller than Gina. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Gina",
"first_ok": true,
"gen": " Gina",
"gold": "Gina",
"prompt": "Finn is taller than Hugo. Hugo is taller than Gina. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Dana",
"first_ok": true,
"gen": " Dana",
"gold": "Dana",
"prompt": "Dana is taller than Hugo. Hugo is taller than Finn. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Dana is taller than Hugo. Hugo is taller than Finn. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Gina",
"first_ok": true,
"gen": " Gina",
"gold": "Gina",
"prompt": "Gina is taller than Dana. Dana is taller than Cara. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Cara",
"first_ok": true,
"gen": " Cara",
"gold": "Cara",
"prompt": "Gina is taller than Dana. Dana is taller than Cara. Of all of them, the shortest is who? Return only the name."
}
],
"first_name_accuracy": 1.0
},
"depth_3": {
"N": 128,
"answer_anywhere_rate": 1.0,
"chance": 0.25,
"ci95_first": [
0.9708620041018919,
0.9999999999999999
],
"examples": [
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Finn is taller than Dana. Dana is taller than Hugo. Hugo is taller than Ben. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Ben",
"first_ok": true,
"gen": " Ben",
"gold": "Ben",
"prompt": "Finn is taller than Dana. Dana is taller than Hugo. Hugo is taller than Ben. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Ben",
"first_ok": true,
"gen": " Ben",
"gold": "Ben",
"prompt": "Ben is taller than Dana. Dana is taller than Evan. Evan is taller than Finn. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Ben is taller than Dana. Dana is taller than Evan. Evan is taller than Finn. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Hugo",
"first_ok": true,
"gen": " Hugo",
"gold": "Hugo",
"prompt": "Hugo is taller than Evan. Evan is taller than Dana. Dana is taller than Gina. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Gina",
"first_ok": true,
"gen": " Gina",
"gold": "Gina",
"prompt": "Hugo is taller than Evan. Evan is taller than Dana. Dana is taller than Gina. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Hugo",
"first_ok": true,
"gen": " Hugo",
"gold": "Hugo",
"prompt": "Hugo is taller than Ben. Ben is taller than Dana. Dana is taller than Finn. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Hugo is taller than Ben. Ben is taller than Dana. Dana is taller than Finn. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Anna",
"first_ok": true,
"gen": " Anna",
"gold": "Anna",
"prompt": "Anna is taller than Dana. Dana is taller than Gina. Gina is taller than Finn. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Anna is taller than Dana. Dana is taller than Gina. Gina is taller than Finn. Of all of them, the shortest is who? Return only the name."
}
],
"first_name_accuracy": 1.0
}
}
}

497
evals/chat_activation_polish900_gpu.json Normal file
View File

@@ -0,0 +1,497 @@
{
"chat_format": "User:\\n...\\nAssistant:\\n",
"ckpt": "runs/v10_348m_cone_sft_polish900/sft_ckpt_0000300.pt",
"code": {
"N": 6,
"ast_rate": 0.16666666666666666,
"body_rate": 0.16666666666666666,
"ci95_exec": [
0.03005258587173032,
0.563509436563646
],
"examples": [
{
"ast_ok": false,
"body_ok": false,
"code": "def add_one(n):\n \"\"\"Return n plus one.\"\"\"\n return n + 1\n",
"exec_ok": false,
"generated": " return n + 1",
"name": "add_one",
"return_ok": false,
"syntax_error": "unindent does not match any outer indentation level (<unknown>, line 3)"
},
{
"ast_ok": true,
"body_ok": true,
"code": "def reverse_string(s):\n \"\"\"Return s reversed.\"\"\"\n return s[::-1]\n",
"exec_ok": true,
"generated": "return s[::-1]",
"name": "reverse_string",
"return_ok": true
},
{
"ast_ok": false,
"body_ok": false,
"code": "def count_vowels(s):\n \"\"\"Return the number of vowels in s.\"\"\"\n return sum(1 for ch in s.lower() if ch in 'aeiou')\n",
"exec_ok": false,
"generated": " return sum(1 for ch in s.lower() if ch in 'aeiou')",
"name": "count_vowels",
"return_ok": false,
"syntax_error": "unindent does not match any outer indentation level (<unknown>, line 3)"
},
{
"ast_ok": false,
"body_ok": false,
"code": "def is_even(n):\n \"\"\"Return True if n is even.\"\"\"\n return n % 2 == 0\n",
"exec_ok": false,
"generated": " return n % 2 == 0",
"name": "is_even",
"return_ok": false,
"syntax_error": "unindent does not match any outer indentation level (<unknown>, line 3)"
},
{
"ast_ok": false,
"body_ok": false,
"code": "def sum_list(items):\n \"\"\"Return the sum of the numbers in items.\"\"\"\n return sum(items)\n",
"exec_ok": false,
"generated": " return sum(items)",
"name": "sum_list",
"return_ok": false,
"syntax_error": "unindent does not match any outer indentation level (<unknown>, line 3)"
},
{
"ast_ok": false,
"body_ok": false,
"code": "def first_item(items):\n \"\"\"Return the first item in a non-empty list.\"\"\"\n return items[0]\n",
"exec_ok": false,
"generated": " return items[0]",
"name": "first_item",
"return_ok": false,
"syntax_error": "unindent does not match any outer indentation level (<unknown>, line 3)"
}
],
"exec_rate": 0.16666666666666666,
"return_rate": 0.16666666666666666
},
"device": "cuda",
"generated_at": "2026-06-22T21:44:35Z",
"math": {
"N": 128,
"answer_anywhere_rate": 0.359375,
"ci95_anywhere": [
0.281465985317146,
0.44547907602919684
],
"examples": [
{
"anywhere": true,
"final": true,
"first": true,
"gen": " 6",
"gold": 6,
"prompt": "Sam has 4 apples. Lily has 2 more apples than Sam. How many apples does Lily have? Return only the number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 3",
"gold": 4,
"prompt": "Mia had 3 red marbles and found 2 blue marbles. Then she gave 1 marble away. Now Mia has how many marbles? Return only the number."
},
{
"anywhere": true,
"final": true,
"first": true,
"gen": " 20",
"gold": 20,
"prompt": "What is 68 - 48? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 224",
"gold": 214,
"prompt": "What is 118 + 96? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 111",
"gold": 113,
"prompt": "What is 109 + 4? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 342",
"gold": 1110,
"prompt": "What is 74 * 15? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 53",
"gold": 55,
"prompt": "What is 97 - 42? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 57",
"gold": 51,
"prompt": "What is 102 - 51? Return only the final number."
},
{
"anywhere": true,
"final": true,
"first": true,
"gen": " 2",
"gold": 2,
"prompt": "What is 22 - 20? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 54",
"gold": 58,
"prompt": "What is 26 + 32? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 7895",
"gold": 1469,
"prompt": "What is 113 * 13? Return only the final number."
},
{
"anywhere": true,
"final": true,
"first": true,
"gen": " 43",
"gold": 43,
"prompt": "What is 43 * 1? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 109",
"gold": 103,
"prompt": "What is 111 - 8? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 150",
"gold": 122,
"prompt": "What is 79 + 43? Return only the final number."
},
{
"anywhere": true,
"final": true,
"first": true,
"gen": " 120",
"gold": 120,
"prompt": "What is 60 * 2? Return only the final number."
},
{
"anywhere": false,
"final": false,
"first": false,
"gen": " 161",
"gold": 151,
"prompt": "What is 72 + 79? Return only the final number."
}
],
"final_numeric_accuracy": 0.359375,
"first_numeric_accuracy": 0.359375
},
"step": 300,
"transitive": {
"depth_1": {
"N": 128,
"answer_anywhere_rate": 1.0,
"chance": 0.5,
"ci95_first": [
0.9708620041018919,
0.9999999999999999
],
"examples": [
{
"anywhere": true,
"first_name": "Dana",
"first_ok": true,
"gen": " Dana",
"gold": "Dana",
"prompt": "Dana is taller than Cara. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Cara",
"first_ok": true,
"gen": " Cara",
"gold": "Cara",
"prompt": "Dana is taller than Cara. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Dana",
"first_ok": true,
"gen": " Dana",
"gold": "Dana",
"prompt": "Dana is taller than Gina. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Gina",
"first_ok": true,
"gen": " Gina",
"gold": "Gina",
"prompt": "Dana is taller than Gina. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Dana",
"first_ok": true,
"gen": " Dana",
"gold": "Dana",
"prompt": "Dana is taller than Hugo. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Hugo",
"first_ok": true,
"gen": " Hugo",
"gold": "Hugo",
"prompt": "Dana is taller than Hugo. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Gina",
"first_ok": true,
"gen": " Gina",
"gold": "Gina",
"prompt": "Gina is taller than Dana. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Dana",
"first_ok": true,
"gen": " Dana",
"gold": "Dana",
"prompt": "Gina is taller than Dana. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Cara",
"first_ok": true,
"gen": " Cara",
"gold": "Cara",
"prompt": "Cara is taller than Finn. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Cara is taller than Finn. Of all of them, the shortest is who? Return only the name."
}
],
"first_name_accuracy": 1.0
},
"depth_2": {
"N": 128,
"answer_anywhere_rate": 1.0,
"chance": 0.3333333333333333,
"ci95_first": [
0.9708620041018919,
0.9999999999999999
],
"examples": [
{
"anywhere": true,
"first_name": "Evan",
"first_ok": true,
"gen": " Evan",
"gold": "Evan",
"prompt": "Evan is taller than Dana. Dana is taller than Anna. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Anna",
"first_ok": true,
"gen": " Anna",
"gold": "Anna",
"prompt": "Evan is taller than Dana. Dana is taller than Anna. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Ben",
"first_ok": true,
"gen": " Ben",
"gold": "Ben",
"prompt": "Ben is taller than Evan. Evan is taller than Anna. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Anna",
"first_ok": true,
"gen": " Anna",
"gold": "Anna",
"prompt": "Ben is taller than Evan. Evan is taller than Anna. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Finn is taller than Hugo. Hugo is taller than Gina. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Gina",
"first_ok": true,
"gen": " Gina",
"gold": "Gina",
"prompt": "Finn is taller than Hugo. Hugo is taller than Gina. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Dana",
"first_ok": true,
"gen": " Dana",
"gold": "Dana",
"prompt": "Dana is taller than Hugo. Hugo is taller than Finn. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Dana is taller than Hugo. Hugo is taller than Finn. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Gina",
"first_ok": true,
"gen": " Gina",
"gold": "Gina",
"prompt": "Gina is taller than Dana. Dana is taller than Cara. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Cara",
"first_ok": true,
"gen": " Cara",
"gold": "Cara",
"prompt": "Gina is taller than Dana. Dana is taller than Cara. Of all of them, the shortest is who? Return only the name."
}
],
"first_name_accuracy": 1.0
},
"depth_3": {
"N": 128,
"answer_anywhere_rate": 1.0,
"chance": 0.25,
"ci95_first": [
0.9708620041018919,
0.9999999999999999
],
"examples": [
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Finn is taller than Dana. Dana is taller than Hugo. Hugo is taller than Ben. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Ben",
"first_ok": true,
"gen": " Ben",
"gold": "Ben",
"prompt": "Finn is taller than Dana. Dana is taller than Hugo. Hugo is taller than Ben. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Ben",
"first_ok": true,
"gen": " Ben",
"gold": "Ben",
"prompt": "Ben is taller than Dana. Dana is taller than Evan. Evan is taller than Finn. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Ben is taller than Dana. Dana is taller than Evan. Evan is taller than Finn. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Hugo",
"first_ok": true,
"gen": " Hugo",
"gold": "Hugo",
"prompt": "Hugo is taller than Evan. Evan is taller than Dana. Dana is taller than Gina. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Gina",
"first_ok": true,
"gen": " Gina",
"gold": "Gina",
"prompt": "Hugo is taller than Evan. Evan is taller than Dana. Dana is taller than Gina. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Hugo",
"first_ok": true,
"gen": " Hugo",
"gold": "Hugo",
"prompt": "Hugo is taller than Ben. Ben is taller than Dana. Dana is taller than Finn. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Hugo is taller than Ben. Ben is taller than Dana. Dana is taller than Finn. Of all of them, the shortest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Anna",
"first_ok": true,
"gen": " Anna",
"gold": "Anna",
"prompt": "Anna is taller than Dana. Dana is taller than Gina. Gina is taller than Finn. Of all of them, the tallest is who? Return only the name."
},
{
"anywhere": true,
"first_name": "Finn",
"first_ok": true,
"gen": " Finn",
"gold": "Finn",
"prompt": "Anna is taller than Dana. Dana is taller than Gina. Gina is taller than Finn. Of all of them, the shortest is who? Return only the name."
}
],
"first_name_accuracy": 1.0
}
}
}

1351
evals/diag_focus600_raw_gpu.json Normal file
View File

File diff suppressed because it is too large Load Diff

1351
evals/diag_polish900_raw_gpu.json Normal file
View File

File diff suppressed because it is too large Load Diff

1351
evals/diag_postsft_sft300_gpu.json Normal file
View File

File diff suppressed because it is too large Load Diff

1351
evals/v10_diagnostic_probe_0210000_gpu_matched_sft300.json Normal file
View File

File diff suppressed because it is too large Load Diff

8
generation_config.json Normal file
View File

@@ -0,0 +1,8 @@
{
"bos_token_id": 0,
"do_sample": false,
"eos_token_id": 0,
"max_new_tokens": 128,
"pad_token_id": 0,
"transformers_version": "5.5.4"
}

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d6d8f6eda578fb8c8437094a2493f88a9529615c94f1911262c2fb277ff31b51
size 1392789664

162881
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

10
tokenizer_config.json Normal file
View File

@@ -0,0 +1,10 @@
{
"backend": "tokenizers",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"is_local": true,
"model_max_length": 1000000000000000019884624838656,
"pad_token": "<|endoftext|>",
"tokenizer_class": "TokenizersBackend",
"unk_token": "<|endoftext|>"
}

210
training_summary.json Normal file
View File

@@ -0,0 +1,210 @@
{
"activation_progression": {
"chat": {
"focus600": {
"code_body_rate": 0.0,
"code_exec_rate": 0.0,
"code_return_rate": 0.0,
"math_answer_anywhere_rate": 0.28125,
"math_final_numeric_accuracy": 0.28125,
"transitive_answer_anywhere_rate": {
"depth_1": 1.0,
"depth_2": 1.0,
"depth_3": 1.0
},
"transitive_first_name_accuracy": {
"depth_1": 1.0,
"depth_2": 1.0,
"depth_3": 1.0
}
},
"polish900": {
"code_body_rate": 0.16666666666666666,
"code_exec_rate": 0.16666666666666666,
"code_return_rate": 0.16666666666666666,
"math_answer_anywhere_rate": 0.359375,
"math_final_numeric_accuracy": 0.359375,
"transitive_answer_anywhere_rate": {
"depth_1": 1.0,
"depth_2": 1.0,
"depth_3": 1.0
},
"transitive_first_name_accuracy": {
"depth_1": 1.0,
"depth_2": 1.0,
"depth_3": 1.0
}
}
},
"raw_completion": {
"base210": {
"code_body_rate": 0.0,
"code_exec_rate": 0.0,
"math_answer_anywhere_rate": 0.25390625,
"math_final_numeric_accuracy": 0.12890625,
"transitive_first_name_accuracy": {
"depth_1": 0.3203125,
"depth_2": 0.2421875,
"depth_3": 0.0546875
}
},
"focus600": {
"code_body_rate": 0.0,
"code_exec_rate": 0.0,
"math_answer_anywhere_rate": 0.23828125,
"math_final_numeric_accuracy": 0.19140625,
"transitive_first_name_accuracy": {
"depth_1": 0.46875,
"depth_2": 0.6171875,
"depth_3": 0.4375
}
},
"polish900": {
"code_body_rate": 0.0,
"code_exec_rate": 0.0,
"math_answer_anywhere_rate": 0.22265625,
"math_final_numeric_accuracy": 0.1796875,
"transitive_first_name_accuracy": {
"depth_1": 0.5390625,
"depth_2": 0.6484375,
"depth_3": 0.53125
}
},
"sft300": {
"code_body_rate": 0.0,
"code_exec_rate": 0.0,
"math_answer_anywhere_rate": 0.23828125,
"math_final_numeric_accuracy": 0.1328125,
"transitive_first_name_accuracy": {
"depth_1": 0.328125,
"depth_2": 0.359375,
"depth_3": 0.125
}
}
}
},
"config_file": {
"path": "runs/v10_348m_cone/_config_run.json",
"sha256": "daf0ca63126bf21513dd6db0f8133106a41e240793e3fbf1587838a133baba5a"
},
"evals": {
"base_raw": {
"path": "runs/v10_348m_cone/evals/v10_diagnostic_probe_0210000_gpu_matched_sft300.json",
"sha256": "a91ba3b55584e5bacc26bbd40a05fe3202a9959a0aa58db78ddbfad82a1823b9"
},
"chat": {
"path": "runs/v10_348m_cone_sft_polish900/evals/chat_activation_polish900_gpu.json",
"sha256": "92577d68fc7793095ccc780918d1cc3643e6658d5cffbdbd1a2bbdae283dd5d7"
},
"focus600_chat": {
"path": "runs/v10_348m_cone_sft_focus600/evals/chat_activation_focus600_gpu.json",
"sha256": "7a9ca095bf17e5ab4b3d2d0a7b0cbd856852f155f9cada7a3057a8f5021cb412"
},
"focus600_raw": {
"path": "runs/v10_348m_cone_sft_focus600/evals/diag_focus600_raw_gpu.json",
"sha256": "1c72d69e624d016868f8561c0df95196bcb680986f8d3b64ba48f1cb0893d868"
},
"raw": {
"path": "runs/v10_348m_cone_sft_polish900/evals/diag_polish900_raw_gpu.json",
"sha256": "0806a1ef2aae347e96e7e5bae6a9e8ca20ed8d1ce7f8eef6bf8146730ee6a9ad"
},
"sft300_raw": {
"path": "runs/v10_348m_cone_sft_210k_clean20k/evals/diag_postsft_sft300_gpu.json",
"sha256": "9ff0306380859a0dda6af17bc67ed1cfe2e5985eef5f978450a51bea281f6f10"
}
},
"format": "Hugging Face Transformers save_pretrained",
"generated_at": "2026-06-23T06:36:05Z",
"metrics": {
"chat": {
"code_note": "Internal manual normalization of single-space body indentation executes 6/6 probe functions.",
"code_strict_exec_rate": 0.16666666666666666,
"code_strict_return_rate": 0.16666666666666666,
"math_answer_anywhere_rate": 0.359375,
"math_final_numeric_accuracy": 0.359375,
"transitive_answer_anywhere_rate": {
"depth_1": 1.0,
"depth_2": 1.0,
"depth_3": 1.0
},
"transitive_first_name_accuracy": {
"depth_1": 1.0,
"depth_2": 1.0,
"depth_3": 1.0
}
},
"raw_completion": {
"code_body_rate": 0.0,
"code_exec_rate": 0.0,
"math_answer_anywhere_rate": 0.22265625,
"math_final_numeric_accuracy": 0.1796875,
"transitive_first_name_accuracy": {
"depth_1": 0.5390625,
"depth_2": 0.6484375,
"depth_3": 0.53125
}
}
},
"model_name": "ConeML 348M Alpha Polish900",
"source_checkpoint": {
"path": "runs/v10_348m_cone_sft_polish900/sft_ckpt_0000300.pt",
"sft_meta": {
"features": 9350,
"final_loss": 0.47538161277770996,
"initial_loss": 0.5603938698768616,
"load_meta": {
"checkpoint": "runs/v10_348m_cone_sft_focus600/sft_ckpt_0000300.pt",
"loaded": true,
"source_step": 300
},
"loss_decreased": true,
"mask_smoke": {
"longer_than_block_size": false,
"record_id": "v10_stage2b_polish_v1_000000_code_body_four_space_indent_add_one_0",
"tokens": 51,
"train_tokens": 6,
"turn_checks": [
{
"role": "user",
"target": false,
"tokens": 38,
"turn": 0
},
{
"role": "assistant",
"target": true,
"tokens": 6,
"turn": 1
}
]
},
"records_seen": 9350,
"signoff": {
"approval_scope": "User requested polish after focus600 showed chat-format transitive 100%, code return bodies with indentation instability, and weak math. Scope: four-space code body formatting, complete function exact format, arithmetic/prose math answer correctness, and transitive retention.",
"approved_by": "user",
"base_ckpt": {
"path": "runs/v10_348m_cone_sft_focus600/sft_ckpt_0000300.pt",
"sha256": "1f117a279128016433331c19e9c95c828aeaaaf0f6d959c759829cd32c6257be"
},
"config_file": {
"path": "runs/v10_348m_cone/_config_run.json",
"sha256": "daf0ca63126bf21513dd6db0f8133106a41e240793e3fbf1587838a133baba5a"
},
"data_files": [
{
"path": "data/corpus/instruction/_staging_sft-selected-v2/generated/v10_stage2b_polish_codegen_math_v1.jsonl",
"records": 9350,
"sha256": "4895bdc8665f2432a2c7aecbbc697a5a350438134fbd882c6f742bcff9f754b5"
}
],
"signoff_file": "notes/signoffs/sft_stage2b_polish_v1_from_focus600.json"
},
"steps": 300
},
"sha256": "2bb974f77936272defee2aa99096a967efaa298ab3868112189daa5254ef26d9",
"step": 300
},
"tokenizer_source": {
"path": "models/tokenizers/v9_67m_32k"
}
}