初始化项目，由ModelHub XC社区提供模型

Model: reaperdoesntknow/LFM2.5-1.2B-Distilled-SFT Source: Original Platform
2026-07-14 15:23:47 +08:00
commit b61ac32af3
8 changed files with 324299 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,296 @@
+---
+library_name: transformers
+pipeline_tag: text-generation
+license: apache-2.0
+language:
+- en
+base_model: liquid/LFM2.5-1.2B-Instruct
+datasets:
+- 0xZee/dataset-CoT-Linear-Algebra-667
+- 0xZee/dataset-CoT-Electromagnetism-580
+- 0xZee/dataset-CoT-Classical-Mechanics-343
+- 0xZee/dataset-CoT-Differential-Equations-636
+- 0xZee/dataset-CoT-mathematics
+- KK04/LogicInference_OA
+tags:
+- causal-lm
+- text-generation
+- distillation
+- knowledge-distillation
+- sft
+- reasoning
+- chain-of-thought
+- mathematics
+- physics
+- stem
+- logic
+- logical-inference
+- hybrid
+- ssm
+- liquid
+- edge
+- convergentintel
+- fp32
+---
+
+# LFM2.5-1.2B-Distilled-SFT
+
+A 1.2B hybrid model (SSM + attention) built in two stages: knowledge distillation from a 24B MoE hybrid teacher on STEM chain-of-thought data, then supervised fine-tuning on logical inference. The first proof-weighted distillation + SFT pipeline on a non-transformer architecture.
+
+Liquid Foundation Models run at 239 tok/s on AMD CPU and fit under 1GB of RAM. This model adds structured STEM reasoning and formal logical inference to that efficiency substrate.
+
+> *"Structure beats scale, collaboration beats hierarchy, observation beats theory."*
+> — Convergent Intelligence LLC: Research Division
+
+## Training Pipeline
+
+### Stage 1: Knowledge Distillation (STEM Reasoning Backbone)
+
+LFM2.5-1.2B distilled from [LFM2-24B-A2B](https://huggingface.co/liquid/LFM2-24B-A2B) — a 24B MoE hybrid (SSM + attention) with only 2B active parameters per token. Teacher and student share the LFM hybrid architecture, so the KL divergence transfers reasoning patterns between architecturally compatible models.
+
+**Data:** 2,802 STEM CoT samples from 5 domains:
+
+| Domain | Samples |
+|---|---|
+| Linear Algebra | 667 |
+| Differential Equations | 636 |
+| Electromagnetism | 580 |
+| Mathematics | 576 |
+| Classical Mechanics | 343 |
+
+All from [0xZee](https://huggingface.co/0xZee). Focused subset — core mathematical reasoning domains that share the most structural overlap with logical inference.
+
+**Loss function:**
+
+1. **Proof-Weighted Cross-Entropy (55%)** — 2.5x → 1.5x on derivation tokens
+2. **Knowledge Distillation KL Divergence (45%)** — T=2.0, scaled by T²
+
+**Training format:**
+
+```
+Solve the following problem carefully and show a rigorous derivation.
+
+Problem:
+{question}
+
+Proof:
+{CoT}
+
+Final Answer:
+{response}
+```
+
+**Stage 1 hyperparameters:**
+
+| Parameter | Value |
+|---|---|
+| Epochs | 1 |
+| Effective batch size | 8 |
+| Learning rate | 1.5e-5 → 1e-6 (cosine) |
+| Temperature | 2.0 |
+| Proof weight | 2.5 → 1.5 |
+| Precision | bf16 |
+
+---
+
+### Stage 2: Logical Inference SFT
+
+Fine-tuned on [KK04/LogicInference_OA](https://huggingface.co/datasets/KK04/LogicInference_OA) — a reproduction of the [LogicInference dataset](https://openreview.net/pdf?id=HAGeIS_Lcg9) (Santiago Ontañón, Google Research) formatted for instruction-following. IID split, LOGICINFERENCEe format (inference first, answer at end). 5,491 unique inference problems extended to ~54,607 instruction-response pairs.
+
+**Why logical inference on a hybrid architecture?** SSM components excel at sequential state propagation — exactly what formal logical inference requires. Each premise updates a logical state, and the conclusion follows from the final state. The hybrid architecture's inductive bias naturally aligns with propositional logic chains. SFT activates this alignment explicitly.
+
+**Training format:**
+
+```
+### Instruction:
+{instruction}
+
+### Response:
+{response}
+```
+
+**Stage 2 hyperparameters:**
+
+| Parameter | Value |
+|---|---|
+| Epochs | 1 |
+| Effective batch size | 8 |
+| Learning rate | 5e-6 (conservative to preserve backbone) |
+| Gradient checkpointing | Enabled |
+| Precision | bf16 |
+
+## Model Details
+
+| Attribute | Value |
+|---|---|
+| **Architecture** | LFM2.5 (hybrid SSM + attention) |
+| **Parameters** | 1.2B |
+| **Base model** | [liquid/LFM2.5-1.2B-Instruct](https://huggingface.co/liquid/LFM2.5-1.2B-Instruct) |
+| **Teacher model** | [liquid/LFM2-24B-A2B](https://huggingface.co/liquid/LFM2-24B-A2B) |
+| **Stage 1 data** | 2,802 STEM CoT samples (5 datasets) |
+| **Stage 2 data** | [KK04/LogicInference_OA](https://huggingface.co/datasets/KK04/LogicInference_OA) (~54,607 pairs) |
+| **Inference** | 239 tok/s AMD CPU, 82 tok/s mobile NPU, sub-1GB RAM |
+| **Context length** | 1024 tokens (training) |
+| **License** | Apache 2.0 |
+| **Developer** | Reaperdoesntrun / [Convergent Intelligence LLC](https://convergentintel.com): Research Division |
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_id = "reaperdoesntknow/LFM2.5-1.2B-Distilled-SFT"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
+    device_map="auto",
+)
+
+# Logical inference (Stage 2)
+prompt = """### Instruction:
+Consider the following premises: If p then q. If q then r. p is true. What can we infer?
+
+### Response:
+"""
+
+# STEM derivation (Stage 1 still works)
+prompt_stem = """Solve the following problem carefully and show a rigorous derivation.
+
+Problem:
+Solve the system of linear equations: 2x + y = 5, x - y = 1.
+
+Proof:
+"""
+
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+with torch.no_grad():
+    outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+### GGUF
+
+Quantized versions at [reaperdoesntknow/LFM2.5-1.2B-Distilled-SFT-GGUF](https://huggingface.co/reaperdoesntknow/LFM2.5-1.2B-Distilled-SFT-GGUF).
+
+## Prompt Formats
+
+**STEM derivation (Stage 1):**
+
+```
+Solve the following problem carefully and show a rigorous derivation.
+
+Problem:
+[Your problem]
+
+Proof:
+```
+
+**Logical inference (Stage 2):**
+
+```
+### Instruction:
+[Your question or logical inference problem]
+
+### Response:
+```
+
+## Intended Uses
+
+**Good for:** On-device logical inference and STEM reasoning, mobile/edge/IoT deployment, formal reasoning tasks, educational tutoring, embedded inference pipelines, anywhere you need structured reasoning under 1GB.
+
+**Not for:** Formal proof verification, safety-critical systems, complex multi-step proofs beyond model capacity, or long-context tasks beyond 1024 tokens.
+
+## Limitations
+
+1.2B hybrid model. The SSM components give excellent inference speed but the model has hard capacity limits. Trained on 2,802 STEM samples (smaller than the 6,122 used for Qwen3 variants). Logical inference strongest on propositional logic patterns in the training data. Complex nested quantifiers may exceed capacity. Always verify.
+
+## Related Models
+
+| Model | Description |
+|---|---|
+| [LFM2.5-1.2B-Distilled](https://huggingface.co/reaperdoesntknow/LFM2.5-1.2B-Distilled) | Stage 1 only — pure STEM backbone |
+| [LFM2.5-1.2B-Distilled-SFT-GGUF](https://huggingface.co/reaperdoesntknow/LFM2.5-1.2B-Distilled-SFT-GGUF) | This model quantized for edge deployment |
+| [Qwen3-1.7B-Coder-Distilled-SFT](https://huggingface.co/reaperdoesntknow/Qwen3-1.7B-Coder-Distilled-SFT) | Transformer variant, Coder teacher + logical inference |
+| [Qwen3-1.7B-Distilled-30B-A3B-SFT](https://huggingface.co/reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B-SFT) | Transformer variant, Instruct teacher + legal SFT |
+
+## Discrepancy Calculus Foundation
+
+This model is part of the [Convergent Intelligence LLC: Research Division](https://huggingface.co/reaperdoesntknow) portfolio. All models in this portfolio are developed under the Discrepancy Calculus (DISC) framework — a measure-theoretic approach to understanding and controlling the gap between what a model *should* produce and what it *actually* produces.
+
+DISC treats training singularities (loss plateaus, mode collapse, catastrophic forgetting) not as failures to be smoothed over, but as **structural signals** that reveal the geometry of the learning problem. Key concepts:
+
+- **Discrepancy Operator (D):** Measures the gap between expected and observed behavior at each training step
+- **Jump Sets:** Boundaries where model behavior changes discontinuously — these are *features*, not bugs
+- **Ghost Imprinting:** Teacher knowledge that transfers to student models through weight-space topology rather than explicit distillation signal
+
+For the full mathematical treatment, see [Discrepancy Calculus: Foundations and Core Theory](https://huggingface.co/reaperdoesntknow/Discrepancy_Calculus) (DOI: 10.57967/hf/8194).
+
+**Citation chain:** [Structure Over Scale](https://huggingface.co/reaperdoesntknow/Structure-Over-Scale) (DOI: 10.57967/hf/8165) → [Three Teachers to Dual Cognition](https://huggingface.co/reaperdoesntknow/DualMind_Methodolgy) (DOI: 10.57967/hf/8184) → [Discrepancy Calculus](https://huggingface.co/reaperdoesntknow/Discrepancy_Calculus) (DOI: 10.57967/hf/8194)
+
+## Citation
+
+```bibtex
+@misc{colca2026lfmsft,
+  title={Hybrid SSM/Attention Distillation + Logical Inference: LFM2-24B to LFM2.5-1.2B},
+  year={2026},
+  publisher={HuggingFace},
+  url={https://huggingface.co/reaperdoesntknow/LFM2.5-1.2B-Distilled-SFT},
+  note={Convergent Intelligence LLC: Research Division}
+}
+```
+
+## References
+
+Santiago Ontañón. "LogicInference: A Large-Scale Dataset for Logical Inference." *ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models.*
+[Paper](https://openreview.net/pdf?id=HAGeIS_Lcg9) | [Code](https://github.com/google-research/google-research/tree/master/logic_inference_dataset)
+
+---
+
+
+<!-- CIX-CROSSLINK-START -->
+
+---
+
+## From the Convergent Intelligence Portfolio
+
+**[DistilQwen Collection](https://huggingface.co/collections/reaperdoesntknow/distilqwen-69bf40ec669117e3f069ef1c)** — Our only BF16 series. Proof-weighted distillation from Qwen3-30B-A3B → 1.7B and 0.6B on H100. Three teacher variants (Instruct, Thinking, Coder), nine models, 2,788 combined downloads. The rest of the portfolio proves structure beats scale on CPU. This collection shows what happens when you give the methodology real hardware.
+
+Top model: [Qwen3-1.7B-Coder-Distilled-SFT](https://huggingface.co/reaperdoesntknow/Qwen3-1.7B-Coder-Distilled-SFT) — 508 downloads
+
+Full methodology: [Structure Over Scale (DOI: 10.57967/hf/8165)](https://doi.org/10.57967/hf/8165)
+
+*Convergent Intelligence LLC: Research Division*
+
+<!-- CIX-CROSSLINK-END -->
+
+*Convergent Intelligence LLC: Research Division*
+*"Where classical analysis fails to see, we begin."*
+
+---
+
+## Convergent Intelligence Portfolio
+
+*Part of the [Liquid Foundation Model Series](https://huggingface.co/reaperdoesntknow) by [Convergent Intelligence LLC: Research Division](https://huggingface.co/reaperdoesntknow)*
+
+
+### Top Models from Our Lab
+
+| Model | Downloads |
+|-------|-----------|
+| [Qwen3-1.7B-Thinking-Distil](https://huggingface.co/reaperdoesntknow/Qwen3-1.7B-Thinking-Distil) | 501 |
+| [Qwen3-1.7B-Coder-Distilled-SFT](https://huggingface.co/reaperdoesntknow/Qwen3-1.7B-Coder-Distilled-SFT) | 302 |
+| [Qwen3-0.6B-Distilled-30B-A3B-Thinking-SFT-GGUF](https://huggingface.co/reaperdoesntknow/Qwen3-0.6B-Distilled-30B-A3B-Thinking-SFT-GGUF) | 203 |
+| [Qwen3-1.7B-Coder-Distilled-SFT-GGUF](https://huggingface.co/reaperdoesntknow/Qwen3-1.7B-Coder-Distilled-SFT-GGUF) | 194 |
+| [Qwen3-1.7B-Distilled-30B-A3B-SFT-GGUF](https://huggingface.co/reaperdoesntknow/Qwen3-1.7B-Distilled-30B-A3B-SFT-GGUF) | 175 |
+
+**Total Portfolio: 41 models | 2,781 total downloads**
+
+
+*Last updated: 2026-03-28 12:47 UTC*
+
+---
+<sub>Part of the [reaperdoesntknow research portfolio](https://huggingface.co/reaperdoesntknow) — 48 models, 12,094 total downloads | Last refreshed: 2026-03-29 21:04 UTC</sub>
+<!-- cix-keeper-ts:2026-07-12T13:15:45Z -->
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,45 @@
+{{- bos_token -}}
+{%- set keep_past_thinking = keep_past_thinking | default(false) -%}
+{%- set ns = namespace(system_prompt="") -%}
+{%- if messages[0]["role"] == "system" -%}
+    {%- set ns.system_prompt = messages[0]["content"] -%}
+    {%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+    {%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: [" -%}
+    {%- for tool in tools -%}
+        {%- if tool is not string -%}
+            {%- set tool = tool | tojson -%}
+        {%- endif -%}
+        {%- set ns.system_prompt = ns.system_prompt + tool -%}
+        {%- if not loop.last -%}
+            {%- set ns.system_prompt = ns.system_prompt + ", " -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- set ns.system_prompt = ns.system_prompt + "]" -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+    {{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- set ns.last_assistant_index = -1 -%}
+{%- for message in messages -%}
+    {%- if message["role"] == "assistant" -%}
+        {%- set ns.last_assistant_index = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+{%- for message in messages -%}
+    {{- "<|im_start|>" + message["role"] + "\n" -}}
+    {%- set content = message["content"] -%}
+    {%- if content is not string -%}
+        {%- set content = content | tojson -%}
+    {%- endif -%}
+    {%- if message["role"] == "assistant" and not keep_past_thinking and loop.index0 != ns.last_assistant_index -%}
+        {%- if "</think>" in content -%}
+            {%- set content = content.split("</think>")[-1] | trim -%}
+        {%- endif -%}
+    {%- endif -%}
+    {{- content + "<|im_end|>\n" -}}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{- "<|im_start|>assistant\n" -}}
+{%- endif -%}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,61 @@
+{
+  "architectures": [
+    "Lfm2ForCausalLM"
+  ],
+  "block_auto_adjust_ff_dim": true,
+  "block_dim": 2048,
+  "block_ff_dim": 12288,
+  "block_ffn_dim_multiplier": 1.0,
+  "block_mlp_init_scale": 1.0,
+  "block_multiple_of": 256,
+  "block_norm_eps": 1e-05,
+  "block_out_init_scale": 1.0,
+  "block_use_swiglu": true,
+  "block_use_xavier_init": true,
+  "bos_token_id": 1,
+  "conv_L_cache": 3,
+  "conv_bias": false,
+  "conv_dim": 2048,
+  "conv_use_xavier_init": true,
+  "dtype": "bfloat16",
+  "eos_token_id": 7,
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "layer_types": [
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv"
+  ],
+  "max_position_embeddings": 128000,
+  "model_type": "lfm2",
+  "norm_eps": 1e-05,
+  "num_attention_heads": 32,
+  "num_heads": 32,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "tie_embedding": true,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.0.0",
+  "use_cache": false,
+  "use_pos_enc": true,
+  "vocab_size": 65536
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": [
+    7
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.0.0"
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df9a186def01ec330773b14db8b391b757d4140e27770e19b8246fea09e4a912
+size 2609133520
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,20 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "is_local": true,
+  "legacy": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {},
+  "pad_token": "<|pad|>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "use_default_system_prompt": false,
+  "use_fast": true
+}