初始化项目，由ModelHub XC社区提供模型

Model: MaliosDark/Isabel-50M
Source: Original Platform

PAPER.md

@@ -0,0 +1,113 @@
+# Isabel-50M: A Tiny Language Model Trained From Scratch for the Edge
+
+**Author:** Malios Dark, Ideoa Labs**Status:** Working technical report (June 2026)
+
+---
+
+## Abstract
+
+We present **Isabel-50M**, a roughly 54M-parameter language model built and trained entirely
+on a single consumer GPU (RTX 3090 Ti), with no base model. Its architecture, byte-level BPE
+tokenizer, and weights are all our own, initialized randomly and trained from scratch. Using a
+two-stage recipe (educational from-scratch pretraining followed by targeted benchmark
+fine-tuning on official train splits) Isabel-50M reaches results competitive with a comparable
+from-scratch model of the same size class that was trained on far more data, while training in
+hours rather than weeks. We report honest results, including the benchmarks where a model of
+this size still falls short.
+
+---
+
+## 1. Introduction
+
+The goal is a genuinely original tiny model (no inherited weights) that is competitive on
+standard small-model benchmarks and runs on the edge, built end to end on one consumer GPU.
+Two design choices proved decisive: training on high-quality educational text rather than
+simple stories, and a short, targeted fine-tune that aligns the model with the exact format
+the benchmarks score.
+
+## 2. Benchmarks
+
+We evaluate with the standard zero-shot, length-normalized multiple-choice protocol on five
+public tasks: HellaSwag, ARC-Easy, ARC-Challenge, PIQA, and an integer-arithmetic benchmark.
+The headline metric is the average across tasks (`acc_norm`).
+
+## 3. Method
+
+**3.1 Tokenizer.** A byte-level BPE tokenizer (32k vocab) trained from scratch on our data, so
+the vocabulary and merges are entirely our own.
+
+**3.2 Architecture.** A standard decoder-only transformer: hidden size 512, 9 layers, 8
+heads, 1024 context. Weights are randomly initialized. There is no base model to declare.
+
+**3.3 From-scratch pretraining.** We pretrain on open, permissively-licensed educational
+English text mixed with our own generated reasoning and arithmetic data (~0.8B tokens).
+Educational text is the single biggest driver of benchmark ability: an earlier run on simple
+story text produced fluent but near-chance benchmark scores, while educational text lifts them
+substantially.
+
+**3.4 Targeted benchmark fine-tuning.** We then fine-tune briefly on the official **train**
+splits of the evaluation tasks (ARC, OpenBookQA, SciQ, QASC, CommonsenseQA), in the same plain
+zero-shot format the harness scores. Only train splits are used, so there is no test
+contamination. This stage is fast (minutes) and directly lifts the multiple-choice scores.
+
+## 4. Results
+
+Zero-shot `acc_norm`, measured on the full public test sets.
+
+| Benchmark | Isabel-50M |
+|---|---|
+| HellaSwag | 28.0 |
+| ARC-Easy | 45.6 |
+| ARC-Challenge | 23.1 |
+| PIQA | 58.5 |
+| Arithmetic | 25.8 |
+| **Average** | **~36.7** |
+
+We note that a small evaluation subsample initially overstated some scores; the numbers here are
+from the full test sets and supersede any earlier preliminary figures.
+
+### 4.1 Position in the ~50M size class
+
+On the public small-model leaderboard, the ~50M parameter class contains 12 models whose average
+scores span roughly 32.5 to 39.0. Isabel-50M sits near the median.
+
+| Position in ~50M class | Average |
+|---|---|
+| Best of class | 39.0 |
+| Class median | ~36.0 |
+| **Isabel-50M** | **36.7** (mid-pack) |
+| Lowest of class | 32.5 |
+
+Isabel-50M outperforms several same-size models and trails the strongest, while being trained
+from scratch on a single consumer GPU in hours. Its relative strengths are PIQA and ARC-Easy; the
+gap to the top of the class is concentrated in hard reasoning (ARC-Challenge and arithmetic).
+
+## 5. Efficiency
+
+Tokenizer training, pretraining, and fine-tuning all run on a single RTX 3090 Ti. A competitive
+checkpoint is reached in a few hours. The practical lesson is that measured throughput on
+commodity hardware is far better than conservative worst-case estimates suggest; we budget from
+the observed rate.
+
+## 6. What is genuinely novel, and what is not
+
+We are precise so the work stays credible. Isabel-50M does **not** introduce a new architecture
+or optimizer; it uses a standard decoder. The contribution is an **empirical recipe and result**:
+a fully from-scratch, single-GPU tiny model that is competitive within hours, the clean finding
+that educational data quality dominates for tiny models, and a fast, contamination-free
+fine-tune that aligns the model to the evaluation format. We present this as a recipe and a
+reproducible result, not a fundamental new method.
+
+## 7. Limitations
+
+At ~54M parameters trained from scratch, hard reasoning is the ceiling. Arithmetic and the
+challenge split of ARC stay near chance and did not improve with several targeted fine-tuning
+experiments. Fluent language and broad knowledge are within reach at this scale; multi-step
+reasoning is not. All numbers are our own local evaluation.
+
+## 8. Conclusion
+
+A single person on a single consumer GPU can train a genuinely original tiny language model that
+is competitive with a much more heavily-trained model of the same size, in hours. The decisive
+factors were the quality of the educational pretraining data and a short, format-aligned
+fine-tune, more than any architectural trick.