初始化项目，由ModelHub XC社区提供模型

Model: Bias-variance-tradeoff/slm-125m-base Source: Original Platform
2026-08-07 07:53:44 +08:00
commit d3961d408b
10 changed files with 81312 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,152 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- llama
+- legal
+- finance
+- small-language-model
+- pretrained-from-scratch
+- vizuara
+datasets:
+- HFforLegal/case-law
+- PleIAs/SEC
+- HuggingFaceFW/fineweb-edu
+metrics:
+- perplexity
+---
+
+# SLM-125M-base
+
+A **125.8M-parameter Llama-style language model pretrained from random weights** on a
+legal + financial corpus. It is a **base completer** (next-token prediction only), not
+an instruction-tuned chatbot — give it the start of a sentence and it continues in the
+legal/financial register.
+
+Built end-to-end on [Modal](https://modal.com) for the **Vizuara "SLM from scratch"
+workshop**, replicating the reference pipeline at
+[Vizuara-AI-Lab/slm-125m-from-scratch](https://github.com/Vizuara-AI-Lab/slm-125m-from-scratch).
+
+- 🕹️ **Live demo:** https://slm-125m-site.vercel.app
+- ⚡ **Inference API:** `https://dharsourav03--slm-125m-inference-web.modal.run` (`/generate`)
+
+| | |
+|---|---|
+| **Parameters** | 125,847,552 (~125.8M, tied embeddings) |
+| **Validation perplexity** | **10.87** (held-out 1% split, ~22.0M tokens) |
+| **Train tokens** | 2.18B (1 epoch) |
+| **Vocab / context** | 16,384 byte-level BPE / 1,024 tokens |
+
+## Intended use
+
+Prompt it with the **opening of a sentence** and it completes the thought in a legal or
+financial style. It is a demonstration of the full from-scratch pipeline (data →
+tokenizer → pretraining), not a production model.
+
+**Not intended for:** factual question answering, chat, instruction following, or any
+high-stakes legal/financial decision-making. At 125M parameters it holds very little
+factual knowledge and will confidently produce plausible-sounding but incorrect content.
+
+## Model architecture
+
+Maps 1:1 to `transformers.LlamaConfig`.
+
+| Component | Value |
+|---|---|
+| Architecture | Llama-style decoder-only transformer |
+| Layers | 12 |
+| Hidden size | 768 |
+| Attention heads | 12 (head dim 64), MHA (kv-heads = 12) |
+| MLP | SwiGLU, inner dim 3072 |
+| Normalization | RMSNorm (pre-norm), eps 1e-5 |
+| Positional encoding | RoPE (theta 10000) |
+| Context length | 1024 tokens |
+| Vocabulary | 16,384 (byte-level BPE), tied embeddings |
+| Attention bias | none |
+
+## Training data
+
+Streamed from HuggingFace, cleaned, deduplicated, and decontaminated. Realized mix
+(by real tokens):
+
+| Source | Content | Share |
+|---|---|---|
+| [HFforLegal/case-law](https://huggingface.co/datasets/HFforLegal/case-law) | US court opinions | ~33% |
+| [PleIAs/SEC](https://huggingface.co/datasets/PleIAs/SEC) | SEC filings (10-K, etc.) | ~39% |
+| [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) | Educational web text | ~28% |
+
+**Pipeline:** stream → deterministic rule-based cleaning (line filters, boilerplate
+strip, repetition/language/OCR gates) → exact + MinHash near-dedup → 13-gram
+decontamination against CaseHOLD / LexGLUE eval sets → 16K byte-level BPE tokenizer →
+pack into 1024-token windows (99/1 train/val split). Final corpus: **2.18B train + 22.0M
+val tokens**.
+
+## Training procedure
+
+| Hyperparameter | Value |
+|---|---|
+| Objective | Next-token cross-entropy (causal LM) |
+| Epochs | 1 (2.18B tokens seen) |
+| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1 |
+| Learning rate | 6e-4 → 6e-5, cosine decay |
+| Warmup | 200M tokens |
+| Gradient clip | 1.0 |
+| Global batch | 524,288 tokens/step (micro-batch 32 × seq 1024, grad-accum 2) |
+| Precision | bf16 |
+| Hardware | 8×H100 (single-node DDP), ~24 min |
+
+> The reference workshop trains ~5 epochs (→ val perplexity 8.50). This checkpoint is a
+> **single-epoch** run (val perplexity 10.87); more epochs over the same fixed corpus
+> lower perplexity further.
+
+## Evaluation
+
+Held-out validation perplexity is `exp(mean token-level cross-entropy)` over the 1%
+val split (21,505 windows, ~22.0M tokens):
+
+**Validation perplexity: 10.87** (val loss 2.3856).
+
+## How to use
+
+The model uses custom special tokens; prepending `<|bos|>` matches how it was served and
+gives the best continuations.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "Bias-variance-tradeoff/slm-125m-base"
+tok = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float32).eval()
+
+prompt = "The plaintiff alleges that the defendant"
+bos = tok.convert_tokens_to_ids("<|bos|>")
+eos = tok.convert_tokens_to_ids("<|eos|>")
+ids = torch.tensor([[bos] + tok.encode(prompt, add_special_tokens=False)])
+
+out = model.generate(
+    ids, max_new_tokens=90, min_new_tokens=40, do_sample=True,
+    temperature=0.8, top_k=50, top_p=0.95, repetition_penalty=1.3,
+    eos_token_id=eos, pad_token_id=eos,
+)
+print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
+```
+
+## Limitations and bias
+
+- **Base model, not aligned** — no instruction tuning, no RLHF, no safety filtering.
+- **Not a knowledge base** — a 125M model stores only a few tens of MB of usable
+  knowledge; it fabricates citations, statutes, and figures. Do not rely on any factual
+  claim it makes.
+- **Domain-skewed** — trained ~72% on US legal + SEC text, so it defaults to that
+  register and reflects the biases of those corpora (US-centric law, corporate finance).
+- **English only.**
+
+## Citation / attribution
+
+Trained from scratch as part of the **Vizuara AI Labs "SLM from scratch" workshop**,
+following the pipeline in
+[Vizuara-AI-Lab/slm-125m-from-scratch](https://github.com/Vizuara-AI-Lab/slm-125m-from-scratch).
--- a/config.json
+++ b/config.json
@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.46.3",
+  "use_cache": true,
+  "vocab_size": 16384
+}
--- a/eval.json
+++ b/eval.json
@@ -0,0 +1,6 @@
+{
+  "val_loss": 2.3856,
+  "val_ppl": 10.87,
+  "val_windows": 21505,
+  "val_tokens": 21999615
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.46.3"
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:444c19cf4daebfaa83226c42cbd61aed5ac673f020521f59e7fe06e22ee2f3a8
+size 503405488
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,35 @@
+{
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|assistant|>",
+    "<|system|>"
+  ],
+  "bos_token": {
+    "content": "<|bos|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eos|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|unk|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,72 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|bos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|eos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|unk|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<|system|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|assistant|>",
+    "<|system|>"
+  ],
+  "bos_token": "<|bos|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|eos|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<|unk|>"
+}
--- a/train_summary.json
+++ b/train_summary.json
@@ -0,0 +1,8 @@
+{
+  "steps": 4156,
+  "tokens": 2178940928,
+  "final_loss": 2.263707995414734,
+  "params": 125848320,
+  "world_size": 8,
+  "minutes": 24.39927451213201
+}