初始化项目,由ModelHub XC社区提供模型
Model: ConeML/coneml-810m-alpha Source: Original Platform
This commit is contained in:
59
eval/EVALUATION_METHODOLOGY.md
Normal file
59
eval/EVALUATION_METHODOLOGY.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# ConeML 810M — evaluation methodology and variant lineage (public record)
|
||||
|
||||
Two released variants share one from-scratch 810M base checkpoint and
|
||||
differ only in supervised fine-tuning; both were evaluated with the same
|
||||
bf16 instruments.
|
||||
|
||||
- `coneml-810m-alpha` — function-writing / mixed-arithmetic profile
|
||||
- `coneml-810m-alpha-arithmetic` — carry/borrow arithmetic variant
|
||||
|
||||
## Methodology
|
||||
|
||||
- **ConeML internal certification**: large-n (300–500 per task family)
|
||||
batteries of unseen instances in the trained prompt formats, greedy
|
||||
decoding, repetition
|
||||
penalty 1.15. Code tasks are scored by EXECUTING the generated function
|
||||
against held-out tests; SQL by executing against a live sqlite database;
|
||||
scripts by captured stdout. No pattern-match scoring for anything
|
||||
execution can decide.
|
||||
- **Behavior probes**: held-out designated-refusal prompts plus in-scope
|
||||
contrast pairs in matching wording; one separately rerun, scripted
|
||||
eight-turn conversation; and an everyday-reasoning screen with manual
|
||||
adjudication. Aggregate results and deterministic representative
|
||||
ConeML-owned rows are public; complete rows are retained privately.
|
||||
- **Public benchmarks**: GSM8K (first 200, zero-shot, strict final-number
|
||||
match) and HumanEval (pass@1, tests executed), identical harness across
|
||||
variants and the pretrained base. Aggregate results are public;
|
||||
benchmark rows are not redistributed.
|
||||
- **Quantized variants are evaluated separately**: each shipped GGUF carries
|
||||
its own result table. The GGUF and bf16 harnesses differ in runtime and
|
||||
stop handling, so score gaps are not attributed solely to quantization.
|
||||
|
||||
## Why two variants
|
||||
|
||||
Under fixed architecture and base checkpoint, modest differences in
|
||||
fine-tuning composition produced large, cross-surface capability
|
||||
tradeoffs (e.g., one build scores 96.8% on held-out carry addition while
|
||||
the other scores 79.0% on held-out two-step word problems and 69.0% on
|
||||
held-out function writing). Rather than presenting one checkpoint as
|
||||
dominant, each variant ships with its measured envelope and a
|
||||
recommendation for which profile
|
||||
fits which pipeline. A checkpoint's profile is measured, never inferred.
|
||||
|
||||
## Shared limitations
|
||||
|
||||
Function-writing is evaluated for single Python functions only — neither
|
||||
variant is a general code model (complete programs, executable SQL, and
|
||||
bash scored near zero for both and are stated out of scope). Both variants
|
||||
refused every designated refusal prompt, including real factual questions;
|
||||
this targeted behavior is not evidence of general epistemic calibration.
|
||||
Multi-step GSM-class reasoning is weak in every variant tested this cycle.
|
||||
|
||||
## Evidence disclosure
|
||||
|
||||
The public folders contain aggregate result tables, deterministic
|
||||
representative examples from ConeML-owned instruments, and SHA-256
|
||||
commitments to the complete row-level evidence. Full proprietary probe
|
||||
rows are retained by ConeML rather than distributed. This preserves the
|
||||
measurement instruments while allowing later disclosure to be checked
|
||||
against the evidence frozen for this release.
|
||||
Reference in New Issue
Block a user